I have a Pyramid web app with fail2ban set up to jail ten consecutive 404 statuses (i.e. bots that probe for vulnerabilities), Sentry error logging and, as far as I know, there are no security vulnerabilities. However, every few days I get a notification of a 502 caused by a null byte attack. This is harmless, but it has become very tiresome and I ignored a bizarre but legitimate human-user–generated 502 status as a result.
A null byte attack in Pyramid, in my set-up, raises a URLDecodeError ('utf-8' codec can't decode byte 0xc0 in position 16: invalid start byte) at the url dispatch level, so is not routed to the notfound_view_config decorated view.
Is there any way to capture %EF/%BF in requests in Pyramid or should I block them in Apache?
Comment by Steve Piercy converted into an Answer:
A search in the Pyramid issue tracker yields several related results. The first hit provides one way to deal with it.
In brief, the view constructor class exception_view_config(ExceptionClass, renderer) captures it behaving like notfound_view_config or forbidden_view_config (which aren't passed declared routes in contrast to view_config).
So the 404 view could look like:
from pyramid.view import notfound_view_config
from pyramid.exceptions import URLDecodeError
from pyramid.view import exception_view_config
#exception_view_config(context=URLDecodeError, renderer='json')
#notfound_view_config(renderer='json')
def notfound_view(request):
request.response.status = 404
return {"status": "error"}
This can be tested by visiting the browser http://0.0.0.0:👾👾/%EF%BF (where 👾👾 is the port served onto).
However, there are two additionally considerations.
It does not play well with the debug toolbar (pyramid.includes = pyramid_debugtoolbar in the local configuration ini file).
Also, an error gets raises if any dynamic attribute like request.path_info gets accessed. So either the response is minimally formatted or request.environ['PATH_INFO'] is assigned a new value before any operation in the view (e.g. usage data etc.).
The view call happens after the debugtoolbar error is raises, however, so the first point still stands even with a request.environ['PATH_INFO'] = 'hacked'.
Bonus
As this is unequivocally an attack, this could be customised to play well with fail2ban to block the hacker IP as described here by using a unique status code, say 418, at the first occurrence.
Related
I have studied the code as follow in Cherrypy web development,
if returnpage != '':
raise cherrypy.InternalRedirect(returnpage)
else:
raise cherrypy.HTTPRedirect("/hqc")
Google doesn't help much in this case after I did some research.
I've checked from cherrypy's __doc__, but the documentation there is very terse.
>>>print(cherrypy.InternalRedirect.__doc__)
Exception raised to switch to the handler for a different URL.
This exception will redirect processing to another path within the site
(without informing the client). Provide the new path as an argument when
raising the exception. Provide any params in the querystring for the new URL.
>>> print(cherrypy.HTTPRedirect.__doc__)
Exception raised when the request should be redirected.
This exception will force a HTTP redirect to the URL or URL's you give it.
The new URL must be passed as the first argument to the Exception,
e.g., HTTPRedirect(newUrl). Multiple URLs are allowed in a list.
If a URL is absolute, it will be used as-is. If it is relative, it is
assumed to be relative to the current cherrypy.request.path_info.
If one of the provided URL is a unicode object, it will be encoded
using the default encoding or the one passed in parameter.
There are multiple types of redirect, from which you can select via the
``status`` argument. If you do not provide a ``status`` arg, it defaults to
303 (or 302 if responding with HTTP/1.0).
Examples::
raise cherrypy.HTTPRedirect("")
raise cherrypy.HTTPRedirect("/abs/path", 307)
raise cherrypy.HTTPRedirect(["path1", "path2?a=1&b=2"], 301)
See :ref:`redirectingpost` for additional caveats.
My questions are:
- Why bother with redirect when you can simply invoke another handler?
- What are some practical senarios for the two redirect exception respectively?
InternalRedirect is only handled in the server side, this means that the client would not be aware of that redirection, because in terms of the HTTP protocol that is mediating the session between the client and the server, nothing changed. By server side I mean ONLY CherryPy will be aware of the rediction, if you have some intermediate server (like an nginx reverse proxy) it would not see anything different.
For example if the client visited a url /page_one and then you used raise InternalRedirect('/page_two'), the client (browser) will receive the content from the /page_two handler in the /page_one url. If you raised a regular HTTPRedirect the server would end the first request with an HTTP status code of 303 (or any other status that you passed to the exception) and a Location header to /page_two. Then is the client who will initiate another request to /page_two, basically everybody will be aware of the redirection (more info about HTTP redirection). Most of the time this is the better alternative.
Additionally you could detect if the request came from a previous InternalRedirect by verifying the cherrypy.request.prev property. It will have the previous cherrypy.request object as its value or None.
For the sake of a possible (maybe not the best example) use of an InternalRedirect, checkout this production/beta example page, in addition I added a tool to prohibit the client to reach to handlers directly.
The client will see a different content in the same page /. Note that the access log that CherryPy generates will log the url of the handler that end up handling the request, in this case you will see /_beta or /_production.
import random
import cherrypy
#cherrypy.tools.register('before_handler')
def private_handler():
"""End the request with HTTP 404 not found if the client
tries to reach the handler directly instead of being
internally redirected from other handler.
"""
if cherrypy.request.prev is None:
raise cherrypy.NotFound()
class MainApp:
#cherrypy.expose
def index(self):
# 50/50 change of receiving production or the new SHINY beta page
use_beta = random.randint(0, 1)
if use_beta:
raise cherrypy.InternalRedirect('/_beta')
else:
raise cherrypy.InternalRedirect('/_production')
#cherrypy.tools.private_handler()
#cherrypy.expose
def _production(self):
return (
"<html>"
"<h2>{}</h2>"
"</html>"
).format(
"Welcome to our awesome site!"
)
#cherrypy.tools.private_handler()
#cherrypy.expose
def _beta(self):
return (
"<html>"
'<h1 style="color: blue">{}</h1>'
"<p>{}</p>"
"</html>"
).format(
"Welcome to our awesome site!",
"Here is our new beta content..."
)
cherrypy.quickstart(MainApp())
I am trying to implement Paul Calhoun's Apache FOP solution for creating PDF's from Xpages (from Notes In 9 #102). I am getting the following java exception when trying to run the xAgent that does the processing --> Can't get a Writer while an OutputStream is already in use
The only changes that I have done from Paul's code was to change the package name. I have isolated when the exception happens to the SSJS line: var jce: DominoXMLFO2PDF = new DominoXMLFO2PDF(); All that line does is instantiate the class, there is no custom constructor. I don't believe it is the code itself, but some configuration issue. The SSJS code is in the beforeRenderResponse event where it should be, I haven't changed anything on the xAgent.
I have copied the jar files from Paul's sample database to mine, I have verified that the build paths are the same between the two databases. Everything compiles fine (after I did all this.) This exception appears to be an xpages only exception.
Here's what's really going on with this error:
XPages are essentially servlets... everything that happens in an XPage is just layers on top of a servlet engine. There are basically two types of data that a servlet can send back to whatever is initiating the connection (e.g. a browser): text and binary.
An ordinary XPage sends text -- specifically, HTML. Some xAgents also send text, such as JSON or XML. In any of these scenarios, however, Domino uses a Java Writer to send the response content, because Writers are optimized for sending Character data.
When we need to send binary content, we use an OutputStream instead, because streams are optimized for sending generic byte data. So if we're sending PDF, DOC/XLS/PPT, images, etc., we need to use a stream, because we're sending binary data, not text.
The catch (as you'll soon see, that's a pun) is that we can only use one per response.
Once any HTTP client is told what the content type of a response is, it makes assumptions about how to process that content. So if you tell it to expect application/pdf, it's expecting to only receive binary data. Conversely, if you tell it to expect application/json, it's expecting to only receive character data. If the response includes any data that doesn't match the promised content type, that nearly always invalidates the entire response.
So Domino in its infinite wisdom protects us from making this mistake by only allowing us to send one or the other in a single request, and throws an exception if we disobey that rule.
Unfortunately... if there's any exception in our code when we're trying to send binary content, Domino wants to report that to the consumer... which tries to invoke the output writer to send HTML reporting that something went wrong. Except we already got a handle on the output stream, so Domino isn't allowed to get a handle on the output writer, because that would violate its own rule against only using one per response. This, in turn, throws the exception you reported, masking the exception that actually caused the problem (in your case, probably a ClassNotFoundException).
So how do we make sure that we see the real problem, and not this misdirection? We try:
try {
/*
* Move all your existing code here...
*/
} catch (e) {
print("Error generating dynamic PDF: " + e.toString());
} finally {
facesContext.responseComplete();
}
There are two reasons this is a preferred approach:
If something goes wrong with our code, we don't let Domino throw an exception about it. Instead, we log it (instead of using print to send it to the console and log, you could also toss it to OpenLog, or whatever your preferred logging mechanism happens to be). This means that Domino doesn't try to report the error to the user, because we've promised that we already reported it to ourselves.
By moving the crucial facesContext.responseComplete() call (which is what ultimately tells Domino not to send any content of its own) to the finally block, this ensures it will get executed. If we left it inside the try block, it would get skipped if an exception occurs, because we'd skip straight to the catch... so even though Domino isn't reporting our exception because we caught it, it still tries to invoke the response writer because we didn't tell it not to.
If you follow the above pattern, and something's wrong with your code, then the browser will receive an incomplete or corrupt file, but the log will tell you what went wrong, rather than reporting an error that has nothing to do with the root cause of the problem.
I almost deleted this question, but decided to answer it myself since there is very little out on google when you search for the exception.
The issue was in the xAgent, there is a line importPackage that was incorrect. Fixing this made everything work. The exception verbage: "Can't get a Writer while an OutputStream is already in use" is quite misleading. I don't know what else triggers this exception, but an alternative description would be "Java class ??yourClass?? not found"
If you found this question, then you likely have the same issue. I would ignore what the exception actually says, and check your package statements throughout your application. The java code will error on its own, but your SSJS that references the java will not error until runtime, focus on that code.
Update the response header after the body can solve this kind of problem, example :
HttpServletResponse response = (HttpServletResponse) facesContext.getExternalContext().getResponse();
response.getWriter().write("<html><body>...</body></html>");
response.setContentType("text/html");
response.setHeader("Cache-Control", "no-cache");
response.setCharacterEncoding("UTF-8");
Let us assume I serve data to colleagues in-office with a small Flask app, and let us also assume that it is a project I am not explicitly 'paid to do' so I don't have all the time in the world to write code.
It has occurred to me in my experimentation with pet projects at home that instead of decorating every last route with #app.route('/some/local/page') that I can do the following:
from flask import Flask, render_template, url_for, redirect, abort
from collections import OrderedDict
goodURLS = OrderedDict([('/index','Home'), ##can be passed to the template
('/about', 'About'), ##to create the navigation bar
('/foo', 'Foo'),
('/bar', 'Bar'), ##hence the use of OrderedDict
('/eggs', 'Eggs'), ##to have a set order for that navibar
('/spam', 'Spam')])
app = Flask(__name__)
#app.route('/<destination>')
def goThere(destination):
availableRoutes = goodURLS.keys():
if "/" + destination in availableRoutes:
return render_template('/%s.html' % destination, goodURLS=goodURLS)
else:
abort(404)
#app.errorhandler(404)
def notFound(e):
return render_template('/notFound.html'), 404
Now all I need to do is update my one list, and both my navigation bar and route handling function are lock-step.
Alternatively, I've written a method to determine the viable file locations by using os.walk in conjunction with file.endswith('.aGivenFileExtension') to locate every file which I mean to make accessible. The user's request can then be compared against the list this function returns (which obviously changes the serveTheUser() function.
from os import path, walk
def fileFinder(directory, extension=".html"):
"""Returns a list of files with a given file extension at a given path.
By default .html files are returned.
"""
foundFilesList = []
if path.exists(directory):
for p, d, files in walk(directory):
for file in files:
if file.endswith(extension):
foundFilesList.append(file)
return foundFilesList
goodRoutes = fileFinder('./templates/someFolderWithGoodRoutes/')
The question is, Is This Bad?
There are many aspects of Flask I'm just not using (mainly because I haven't needed to know about them yet) - so maybe this is actually limiting, or redundant when compared against a built-in feature of Flask. Does my lack of explicitly decorating each route rob me of a great feature of Flask?
Additionally, is either of these methods more or less safe than the other? I really don't know much about web security - and like I said, right now this is all in-office stuff, the security of my data is assured by our IT professional and there are no incoming requests from outside the office - but in a real-world setting, would either of these be detrimental? In particular, if I am using the backend to os.walk a location on the server's local disk, I'm not asking to have it abused by some ne'er-do-well am I?
EDIT: I've offered this as a bounty, because if it is not a safe or constructive practice I'd like to avoid using it for things that I'd want to like push to Heroku or just in general publicly serve for family, etc. It just seems like decorating every viable route with app.route is a waste of time.
There isn't anything really wrong with your solution, in my opinion. The problem is that with this kind of setup the things you can do are pretty limited.
I'm not sure if you simplified your code to show here, but if all you are doing in your view function is to gather some data and then select one of a few templates to render it then you might as well render the whole thing in a single page and maybe use a Javascript tab control to divide it up in sections on the client.
If each template requires different data, then the logic that obtains and processes the data for each template will have to be in your view function, and that is going to look pretty messy because you'll have a long chain of if statements to handle each template. Between that and separate view functions per template I think the latter will be quicker, even more so if you also consider the maintenance effort.
Update: based on the conversion in the comments I stand by my answer, with some minor reservations.
I think your solution works and has no major problems. I don't see a security risk because you are validating the input that comes from the client before you use it.
You are just using Flask to serve files that can be considered static if you ignore the navigation bar at the top. You should consider compiling the Flask app into a set of static files using an extension like Frozen-Flask, then you just host the compiled files with a regular web server. And when you need to add/remove routes you can modify the Flask app and compile it again.
Another thought is that your Flask app structure will not scale well if you need to add server-side logic. Right now you don't have any logic in the server, everything is handled by jQuery in the browser, so having a single view function works just fine. If at some point you need to add server logic for these pages then you will find that this structure isn't convenient.
I hope this helps.
I assume based on your code that all the routes have a corresponding template file of the same name (destination to destination.html) and that the goodURL menu bar is changed manually. An easier method would be to try to render the template at request and return your 404 page if it doesn't exist.
from jinja2 import TemplateNotFound
from werkzeug import secure_filename
....
#app.route('/<destination>')
def goThere(destination):
destTemplate = secure_filename("%s.html" % destination)
try:
return render_template(destTemplate, goodURLS=goodURLS)
except TemplateNotFound:
abort(404)
#app.errorhandler(404)
def notFound(e):
return render_template('/notFound.html'), 404
This is adapted from the answer to Stackoverflow: How do I create a 404 page?.
Edit: Updated to make use of Werkzeug's secure_filename to clean user input.
My goal is to create an IIS Managed Module that looks at the Request and filters out content from the POST (XSS attacks, SQL injection, etc).
I'm hung up right now, however, on the process of actually filtering the Request. Here's what I've got so far:
In the Module's Init, I set HttpApplication.BeginRequest to a local event handler. In that event handler, I have the following lines set up:
if (application.Context.Request.HttpMethod == "POST")
{
application.Context.Request.Filter = new HttpRequestFilter(application.Context.Request.Filter);
}
I also set up an HttpResponseFilter on the application.Context.Response.Filter
HttpRequestFilter and HttpResponseFilter are implementations of Stream.
In the response filter, I have the following set up (an override of Stream.Write):
public override void Write(byte[] buffer, int offset, int count)
{
var Content = UTF8Encoding.UTF8.GetString(buffer);
Content = ResponseFilter.Filter(Content);
_responseStream.Write(UTF8Encoding.UTF8.GetBytes(Content), offset, UTF8Encoding.UTF8.GetByteCount(Content));
}
ResponseFilter.Filter is a simple String.Replace, and it does, in fact, replace text correctly.
In the request filter, however, there are 2 issues.
The code I have currently in the RequestFilter (an override of Stream.Read):
public override int Read(byte[] buffer, int offset, int count)
{
var Content = UTF8Encoding.UTF8.GetString(buffer);
Content = RequestFilter.Filter(Content);
if (buffer[0]!= 0)
{
return _requestStream.Read(UTF8Encoding.UTF8.GetBytes(Content), offset, UTF8Encoding.UTF8.GetByteCount(Content));
}
return _requestStream.Read(buffer, offset, count);
}
There are 2 issues with this. First, the filter is called twice, not once, and one of the requests is just basically a stream of /0's. (the if check on buffer[0] filters this currently, but I think that I'm setting something up wrong)
Second, even though I am correctly grabbing content with the .GetString in the read, and then altering it in RequestFilter.Filter(a glorified string.replace()), when I return the byte encoded Content inside the if statement, the input is unmodified.
Here's what I'm trying to figure out:
1) Is there something I can check prior to the filter to ensure that what I'm checking is only the POST and not the other time it is being called? Am I not setting the Application.Context.Request.Filter up correctly?
2) I'm really confused as to why rewriting things to the _requestStream (the HttpApplication.Context.Request.Filter that I sent to the class) isn't showing up. Any input as to something I'm doing wrong would be really appreciated.
Also, is there any difference between HttpApplication.Request and HttpApplication.Context.Request?
edit: for more information, I'm testing this on a simple .aspx page that has a text box, a button and a label, and on button click assigns the text box text to the label's text. Ideally, if I put content in the textbox that should be filtered, it is my understanding that by intercepting and rewriting the post, I can cause the stuff to hit the server as modified. I've run test though with breakpoints in the module and in code, and the module completes before the code behind on the .aspx page is hit. The .aspx page gets the values as passed from the form, and ignores any filtering I attempted to do.
There's a few issues going on here, but for future reference, what explains the page receiving the unfiltered post, as well as the filter being evaluated twice is that you are likely accessing the request object in some way PRIOR to you setting the Request.Filter. This may cause it to evaluate the inputstream, running the currently set filter chain as is, and returning that stream.
For example, simply accessing Request.Form["something"] would cause it to evaluate the inputstream, running the entire filter chain, at that point in time. Any modification to the Request.Filters after this point in time would have no effect, and would appear that this filter is being ignored.
What you wanted to do is possible, but also ASP.NET provides Request Validation to address some of these issues (XSS). However, Sql Injection is usually averted by never constructing queries through string concatenation, not via input sanitizing, though defense-in-depth is usually a good idea.
I have very little to go on here. I can't reproduce this locally, but when users get the error I get an automatic email exception notification:
Invalid length for a Base-64 char array.
at System.Convert.FromBase64String(String s)
at System.Web.UI.ObjectStateFormatter.Deserialize(String inputString)
at System.Web.UI.ObjectStateFormatter.System.Web.UI.IStateFormatter.Deserialize(String serializedState)
at System.Web.UI.Util.DeserializeWithAssert(IStateFormatter formatter, String serializedState)
at System.Web.UI.HiddenFieldPageStatePersister.Load()
I'm inclined to think there is a problem with data that is being assigned to viewstate.
For example:
List<int> SelectedActionIDList = GetSelectedActionIDList();
ViewState["_SelectedActionIDList"] = SelectedActionIDList;
It's difficult to guess the source of the error without being able to reproduce the error locally.
If anyone has had any experience with this error, I would really like to know what you found out.
After urlDecode processes the text, it replaces all '+' chars with ' ' ... thus the error. You should simply call this statement to make it base 64 compatible again:
sEncryptedString = sEncryptedString.Replace(' ', '+');
I've seen this error caused by the combination of good sized viewstate and over aggressive content-filtering devices/firewalls (especially when dealing with K-12 Educational institutions).
We worked around it by storing Viewstate in SQL Server. Before going that route, I would recommend trying to limit your use of viewstate by not storing anything large in it and turning it off for all controls which do not need it.
References for storing ViewState in SQL Server:
MSDN - Overview of PageStatePersister
ASP Alliance - Simple method to store viewstate in SQL Server
Code Project - ViewState Provider Model
My guess is that something is either encoding or decoding too often - or that you've got text with multiple lines in.
Base64 strings have to be a multiple of 4 characters in length - every 4 characters represents 3 bytes of input data. Somehow, the view state data being passed back by ASP.NET is corrupted - the length isn't a multiple of 4.
Do you log the user agent when this occurs? I wonder whether it's a badly-behaved browser somewhere... another possibility is that there's a proxy doing naughty things. Likewise try to log the content length of the request, so you can see whether it only happens for large requests.
Try this:
public string EncodeBase64(string data)
{
string s = data.Trim().Replace(" ", "+");
if (s.Length % 4 > 0)
s = s.PadRight(s.Length + 4 - s.Length % 4, '=');
return Encoding.UTF8.GetString(Convert.FromBase64String(s));
}
int len = qs.Length % 4;
if (len > 0) qs = qs.PadRight(qs.Length + (4 - len), '=');
where qs is any base64 encoded string
As others have mentioned this can be caused when some firewalls and proxies prevent access to pages containing a large amount of ViewState data.
ASP.NET 2.0 introduced the ViewState Chunking mechanism which breaks the ViewState up into manageable chunks, allowing the ViewState to pass through the proxy / firewall without issue.
To enable this feature simply add the following line to your web.config file.
<pages maxPageStateFieldLength="4000">
This should not be used as an alternative to reducing your ViewState size but it can be an effective backstop against the "Invalid length for a Base-64 char array" error resulting from aggressive proxies and the like.
This isn't an answer, sadly. After running into the intermittent error for some time and finally being annoyed enough to try to fix it, I have yet to find a fix. I have, however, determined a recipe for reproducing my problem, which might help others.
In my case it is SOLELY a localhost problem, on my dev machine that also has the app's DB. It's a .NET 2.0 app I'm editing with VS2005. The Win7 64 bit machine also has VS2008 and .NET 3.5 installed.
Here's what will generate the error, from a variety of forms:
Load a fresh copy of the form.
Enter some data, and/or postback with any of the form's controls. As long as there is no significant delay, repeat all you like, and no errors occur.
Wait a little while (1 or 2 minutes maybe, not more than 5), and try another postback.
A minute or two delay "waiting for localhost" and then "Connection was reset" by the browser, and global.asax's application error trap logs:
Application_Error event: Invalid length for a Base-64 char array.
Stack Trace:
at System.Convert.FromBase64String(String s)
at System.Web.UI.ObjectStateFormatter.Deserialize(String inputString)
at System.Web.UI.Util.DeserializeWithAssert(IStateFormatter formatter, String serializedState)
at System.Web.UI.HiddenFieldPageStatePersister.Load()
In this case, it is not the SIZE of the viewstate, but something to do with page and/or viewstate caching that seems to be biting me. Setting <pages> parameters enableEventValidation="false", and viewStateEncryption="Never" in the Web.config did not change the behavior. Neither did setting the maxPageStateFieldLength to something modest.
Take a look at your HttpHandlers. I've been noticing some weird and completely random errors over the past few months after I implemented a compression tool (RadCompression from Telerik). I was noticing errors like:
System.Web.HttpException: Unable to validate data.
System.Web.HttpException: The client disconnected.---> System.Web.UI.ViewStateException: Invalid viewstate.
and
System.FormatException: Invalid length for a Base-64 char array.
System.Web.HttpException: The client disconnected. ---> System.Web.UI.ViewStateException: Invalid viewstate.
I wrote about this on my blog.
This is because of a huge view state, In my case I got lucky since I was not using the viewstate. I just added enableviewstate="false" on the form tag and view state went from 35k to 100 chars
During initial testing for Membership.ValidateUser with a SqlMembershipProvider, I use a hash (SHA1) algorithm combined with a salt, and, if I changed the salt length to a length not divisible by four, I received this error.
I have not tried any of the fixes above, but if the salt is being altered, this may help someone pinpoint that as the source of this particular error.
As Jon Skeet said, the string must be multiple of 4 bytes. But I was still getting the error.
At least it got removed in debug mode. Put a break point on Convert.FromBase64String() then step through the code. Miraculously, the error disappeared for me :) It is probably related to View states and similar other issues as others have reported.
In addition to #jalchr's solution that helped me, I found that when calling ATL::Base64Encode from a c++ application to encode the content you pass to an ASP.NET webservice, you need something else, too. In addition to
sEncryptedString = sEncryptedString.Replace(' ', '+');
from #jalchr's solution, you also need to ensure that you do not use the ATL_BASE64_FLAG_NOPAD flag on ATL::Base64Encode:
BOOL bEncoded = Base64Encode(lpBuffer,
nBufferSizeInBytes,
strBase64Encoded.GetBufferSetLength(base64Length),
&base64Length,ATL_BASE64_FLAG_NOCRLF/*|ATL_BASE64_FLAG_NOPAD*/);