I want to block user-agents with UrlScan on IIS 6. However I am not able to specify a user-agent with a semicolon in the string. I think this is a very common scenario, but I can't find any answer on how to escape a semicolon in UrlScan.ini (where semicolon are used for commenting).
This is the rule:
RuleList=DenyUserAgent
[DenyUserAgent]
DenyDataSection=AgentStrings
ScanHeaders=User-Agent
[AgentStrings]
Mozilla/5.0 (Windows NT 5.1; rv:6.0) Gecko/20100101 Firefox/10.0.2
I tested it and it blocks all the user-agents that start with "Mozilla/5.0 (Windows NT 5.1" because it considers the rest of the string as a comment.
It seems like you would need to URL escape it, i.e. %3B
See http://learn.iis.net/page.aspx/476/common-urlscan-scenarios/ for examples; one such example they have is blocking semicolon in the querystring to block a SQL injection attack
Related
Quite a stupid question but I can't figure out how to do this....
I get the OS name from Get-ADComputer, lets say: "Windows Server 2012 R2 Standard".
I am trying to just select the Year within the Build, eg:
"Windows Server 2012 R2 Standard" > 2012
"Windows Server 2019" > 2019
"Windows Server 2008" > 2008
"Windows Server 2008 R2" > 2008
Quite ignorant on Regex...how could I do it?.
Also, is there a good tutorial on Regex for Powershell you can refer me to?
You can match on the string with the following pattern like so:
$year = if( "Windows Server 2019" -match '\d{4}' ){
$matches[0]
}
This works by using the -match operator to check that a match against a pattern (regex) is found. If the match is found (-match returns $True or $False), you can then use the automatic $matches array to find what value matches the pattern. $matches[0] will contain the matched value, any additional elements would signify matching capturing groups as part of more complex expressions.
As for the expression itself, \d indicates to match on a numeric character and the {4} is a quantifier, stating that we want to match on exactly 4 of the preceeding character or token.
Of course, if -match returned $False above, then $year would not be set with a value.
Note that the -match operator does not support global matching if you ever needed to find all occurrences of a pattern in a string.
But what if I need to match on all occurrences of a pattern in a string?
Your use case above doesn't call for this, but if we do need a global matching option, we can use the [Regex]::Matches method here, which performs a global pattern match on the input string:
$myString = "2019 2018 a clockwork orangutan 2017 2016"
[Regex]::Matches( $myString, '\d{4}' ).Value # => 2019
# 2018
# 2017
# 2016
[Regex]::Matches returns an array of System.Text.RegularExpressions.Match objects. The Value property contains the actual string that matches the pattern, but you can also look at other properties of this object to learn more information as well, such as the which index in the string the pattern match was found in, for example.
A good regex resource
As for "a good tutorial" on regex, I can't recommend one, but I do make use of https://regexr.com/ when writing and testing regular expressions. It doesn't support .NET regex, but you can change the mode to PCRE which works very closely to how .NET regex works. As you write your expressions, it will break your expression down and explain what each character (or special token) means. There is also a decent reference you can use as well.
I am trying to redirect a path e.g. www.something.com/apple/pie to www.something.com/tickets/pie-details
This is what I have tried but doesn't work:
if (req.url ~ "^/apple/.*") {
set req.url = "^/tickets/.*-details";
error 701 req.url;
}
Am I missing something?
You either need to capture the matches from the regex, or if it is simple just replace using regsub()
However I have read that these are no longer bundled in core varnish so you might need a vmod. This one appears to be the one you need: https://gitlab.com/uplex/varnish/libvmod-re
Here are some docs on how this can be used: https://docs.fastly.com/en/guides/vcl-regular-expression-cheat-sheet#capturing-matches
Basically the re object lets you use the matched portion to then assemble the new url using string operations.
All of the above is speculative using my knowledge of vcl and regex, but I personally have not tried it.
Within Node.js, I am using querystring.stringify() to encode an object into a query string for usage in a URL. Values that have spaces are encoded as %20.
I'm working with a particularly finicky web service that will only accept spaces encoded as +, as used to be commonly done prior to RFC3986.
Is there a way to set an option for querystring so that it encodes spaces as +?
Currently I am simply doing a .replace() to replace all instances of %20 with +, but this is a bit tedious if there is an option I can set ahead of time.
If anyone still facing this issue, "qs" npm package has feature to encode spaces as +
qs.stringify({ a: 'b c' }, { format : 'RFC1738' })
I can't think of any library doing that by default, and unfortunately, I'd say your implementation may be the more efficient way to do this, since any other option would probably either do what you're already doing, or will use slower non-compiled pure JavaScript code.
What about asking the web service provider to follow the RFC?
https://github.com/kvz/phpjs is a node.js package that provides all the php functions. The http_build_query implementation at the time of writing this only supports urlencode (the query string includes + instead of spaces), but hopefully soon will include the enc_type parameter / rawurlencode (%20's for spaces).
See http://php.net/http_build_query.
RFC1738 (+'s) will be the default enc_type either way, so you can use it immediately for your purposes.
About the system
I have URLs of this format in my project:-
http://project_name/browse_by_exam/type/tutor_search/keyword/class/new_search/1/search_exam/0/search_subject/0
Where keyword/class pair means search with "class" keyword.
I have a common index.php file which executes for every module in the project. There is only a rewrite rule to remove the index.php from URL:-
RewriteCond $1 !^(index\.php|resources|robots\.txt)
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php [L,QSA]
I am using urlencode() while preparing the search URL and urldecode() while reading the search URL.
Problem
Only the forward slash character is breaking URLs causing 404 page not found error.
For example, if I search one/two the URL is
http://project_name/browse_by_exam/type/tutor_search/keyword/one%2Ftwo/new_search/1/search_exam/0/search_subject/0/page_sort/
How do I fix this? I need to keep index.php hidden in the URL. Otherwise, if that was not needed, there would have been no problem with forward slash and I could have used this URL:-
http://project_name/index.php?browse_by_exam/type/tutor_search/keyword/one
%2Ftwo/new_search/1/search_exam/0/search_subject/0
Apache denies all URLs with %2F in the path part, for security reasons: scripts can't normally (ie. without rewriting) tell the difference between %2F and / due to the PATH_INFO environment variable being automatically URL-decoded (which is stupid, but a long-standing part of the CGI specification so there's nothing can be done about it).
You can turn this feature off using the AllowEncodedSlashes directive, but note that other web servers will still disallow it (with no option to turn that off), and that other characters may also be taboo (eg. %5C), and that %00 in particular will always be blocked by both Apache and IIS. So if your application relied on being able to have %2F or other characters in a path part you'd be limiting your compatibility/deployment options.
I am using urlencode() while preparing the search URL
You should use rawurlencode(), not urlencode() for escaping path parts. urlencode() is misnamed, it is actually for application/x-www-form-urlencoded data such as in the query string or the body of a POST request, and not for other parts of the URL.
The difference is that + doesn't mean space in path parts. rawurlencode() will correctly produce %20 instead, which will work both in form-encoded data and other parts of the URL.
Replace %2F with %252F after url encoding
PHP
function custom_http_build_query($query=array()){
return str_replace('%2F','%252F', http_build_query($query));
}
Handle the request via htaccess
.htaccess
RewriteCond %{REQUEST_URI} ^(.*?)(%252F)(.*?)$ [NC]
RewriteRule . %1/%3 [R=301,L,NE]
Resources
http://www.leakon.com/archives/865
In Apache, AllowEncodedSlashes On would prevent the request from being immediately rejected with a 404.
Just another idea on how to fix this.
$encoded_url = str_replace('%2F', '/', urlencode($url));
I had the same problem with slash in url get param, in my case following php code works:
$value = "hello/world"
$value = str_replace('/', '/', $value;?>
$value = urlencode($value);?>
# $value is now hello%26%2347%3Bworld
I first replace the slash by html entity and then I do the url encoding.
Here's my humble opinion. !!!! Don't !!!! change settings on the server to make your parameters work correctly. This is a time bomb waiting to happen someday when you change servers.
The best way I have found is to just convert the parameter to base 64 encoding. So in my case, I'm calling a php service from Angular and passing a parameter that could contain any value.
So my typescript code in the client looks like this:
private encodeParameter(parm:string){
if (!parm){
return null;
}
return btoa(parm);
}
And to retrieve the parameter in php:
$item_name = $request->getAttribute('item_name');
$item_name = base64_decode($item_name);
On my hosting account this problem was caused by a ModSecurity rule that was set for all accounts automatically. Upon my reporting this problem, their admin quickly removed this rule for my account.
Use a different character and replace the slashes server side
e.g. Drupal.org uses %21 (the excalamation mark character !) to represent the slash in a url parameter.
Both of the links below work:
https://api.drupal.org/api/drupal/includes%21common.inc/7
https://api.drupal.org/api/drupal/includes!common.inc/7
If you're worried that the character may clash with a character in the parameter then use a combination of characters.
So your url would be
http://project_name/browse_by_exam/type/tutor_search/keyword/one_-!two/new_search/1/search_exam/0/search_subject/0
change it out with js and convert it back to a slash server side.
is simple for me use base64_encode
$term = base64_encode($term)
$url = $youurl.'?term='.$term
after you decode the term
$term = base64_decode($['GET']['term'])
this way encode the "/" and "\"
A standard solution for this problem is to allow slashes by making the parameter that may contain slashes the last parameter in the url.
For a product code url you would then have...
mysite.com/product/details/PR12345/22
For a search term you'd have
http://project/search_exam/0/search_subject/0/keyword/Psychology/Management
(The keyword here is Psychology/Management)
It's not a massive amount of work to process the first "named" parameters then concat the remaining ones to be product code or keyword.
Some frameworks have this facility built in to their routing definitions.
This is not applicable to use case involving two parameters that my contain slashes.
I use javascript encodeURI() function for the URL part that has forward slashes that should be seen as characters instead of http address.
Eg:
"/api/activites/" + encodeURI("?categorie=assemblage&nom=Manipulation/Finition")
see http://www.w3schools.com/tags/ref_urlencode.asp
I solved this by using 2 custom functions like so:
function slash_replace($query){
return str_replace('/','_', $query);
}
function slash_unreplace($query){
return str_replace('_','/', $query);
}
So to encode I could call:
rawurlencode(slash_replace($param))
and to decode I could call
slash_unreplace(rawurldecode($param);
Cheers!
You can use %2F if using it this way:
?param1=value1¶m2=value%2Fvalue
but if you use /param1=value1/param2=value%2Fvalue it will throw an error.
My ColdFusion (MX7 on IIS 6) site has search functionality which appends the search term to the URL e.g. http://www.example.com/search.cfm/searchterm.
The problem I'm running into is this is a multilingual site, so the search term may be in another language e.g. القاهرة leading to a search URL such as http://www.example.com/search.cfm/القاهرة
The problem is when I come to retrieve the search term from the URL. I'm using cgi.PATH_INFO to retrieve the path of the search page and the search term and extracting the search term from this e.g. /search.cfm/searchterm however, when unicode characters are used in the search they are converted to question marks e.g. /search.cfm/??????.
These appear actual question marks, rather than the browser not being able to format unicode characters, or them being mangled on output.
I can't find any information about whether ColdFusion supports unicode in the URL, or how I can go about resolving this and getting hold of the complete URL in some way - does anyone have any ideas?
Cheers,
Tom
Edit: Further research has lead me to believe the issue may related to IIS rather than ColdFusion, but my original query still stands.
Further edit
The result of GetPageContext().GetRequest().GetRequestUrl().ToString() is http://www.example.com/search.cfm/searchterm/????? so it appears the issue goes fairly deep.
Yeah, it's not really ColdFusion's fault. It's a common problem.
It's mostly the fault of the original CGI specification, which specifies that PATH_INFO has to be %-decoded, thus losing the original %xx byte sequences that would have allowed you to work out which real characters were meant.
And it's partly IIS's fault, because it always tries to read submitted %xx bytes in the path part as UTF-8-encoded Unicode (unless the path isn't a valid UTF-8 byte sequence in which case it plumps for the Windows default code page, but gives you no way to find out this has happened). Having done so, it puts it in environment variables as a Unicode string (as envvars are Unicode under Windows).
However most byte-based tools using the C stdio (and I'm assuming this applies to ColdFusion, as it does under Perl, Python 2, PHP etc.) then try to read the environment variables as bytes, and the MS C runtime encodes the Unicode contents again using the Windows default code page. So any characters that don't fit in the default code page are lost for good. This would include your Arabic characters when running on a Western Windows install.
A clever script that has direct access to the Win32 GetEnvironmentVariableW API could call that to retrieve a native-Unicode environment variable which they could then encode to UTF-8 or whatever else they wanted, assuming that the input was also UTF-8 (which is what you'd generally want today). However, I don't think CodeFusion gives you this access, and in any case it only works from IIS6 onwards; IIS5.x will throw away any non-default-codepage characters before they even reach the environment variables.
Otherwise, your best bet is URL-rewriting. If a layer above CF can convert that search.cfm/القاهرة to search.cfm/?q=القاهرة then you don't face the same problem, as the QUERY_STRING variable, unlike PATH_INFO, is not specified to be %-decoded, so the %xx bytes remain where a tool at CF's level can see them.
Here's what you could do:
<cfset url.searchTerm = URLEncodedFormat("القاهر", "utf-8") >
<cfset myVar = URLDecode(url.searchTerm , "utf-8") >
Ofcourse, I'd recommend that you work with something like this in that case:
yourtemplate.cfm?searchTerm=%C3%98%C2%A7%C3%99%E2%80%9E
And then you do URL rewriting in IIS (if not already done by framework/rest of the app) http://learn.iis.net/page.aspx/461/creating-rewrite-rules-for-the-url-rewrite-module/ to match your pattern.
You can set the character encoding of the URL and FORM scope using the setEncoding() function:
http://www.adobe.com/livedocs/coldfusion/7/htmldocs/wwhelp/wwhimpl/common/html/wwhelp.htm?context=ColdFusion_Documentation&file=00000623.htm
You need to do this before you access any of the variables in this scope.
But, the default encoding of those scopes is already UTF-8, so this may not help. Also, this would probably not affect the CGI scope.
Is the IIS Server logging the correct characters into the request log?