When performing NTLM authentication sequence in HTTP communications, Windows API function called InitializeSecureContext is used. It includes fContextReq parameter, which accepts the set of flags that regulate the exact behavior of the library in regards to handling challenges and creating responses.
For years we've been using ISC_REQ_INTEGRITY | с ISC_REQ_CONNECTION, and this worked fine for most servers. Now with McAfee Web proxy this combination of flags produces the INVALID_TOKEN error in InitializeSecurityContext on the second step. Removing ISC_REQ_INTEGRITY flag helps with McAfee server (not tested with ISA yet).
The question is whether the flag was used incorrectly (and McAfee proxy is correct by not using it on its side) OR this is a glitch of McAfee proxy OR the set of flags are to be prearranged between parties by design? I understand that in different scenarios there are different flags to be used, and my question is specifically about NTLM in HTTP authentication, where the other party and its capabilities and used flags are not known beforehand.
Related
Background
I've got an app using the requests module to handle connecting to a remote webserver. This works perfectly, but I want to deploy it at within an organisation using an enterprise proxy server. The machines in the organisation have the proxy configured at the operating system level (ie windows setting the system proxy).
I'd prefer to have my app automatically use the already configured OS proxy settings, rather than have to ask them for the info (especially as they use basic authentication, so I'd have to securely store a username/password, not just the proxy host/port).
Question
Does Requests automatically use the operating system's proxy settings if you do not specify a proxy directly yourself?
I couldn't find the definitive answer to this after reading Request's documentation, or the underlying urllib3.
On my dev machine I don't have a proxy to test with, and so would like to know the answer before I go and code manual proxy handling in my app that might not actually be necessary...
Some more info
As a bit of comparison, Urllib does do this - see https://docs.python.org/3/library/urllib.request.html#urllib.request.ProxyHandler ...if no proxy is specified it will utilize the system configured one.
If seemed on my initial review of Request's documentation it didn't use the system configuration, instead only using environment variables if they were set: https://2.python-requests.org/en/master/user/advanced/#proxies
But, after a bit more digging, I found a way to at least obtain the OS proxy configuration, using urllib.request.getproxies(): https://stackoverflow.com/a/16311657/9423009
At this point I thought I'd at least be able to use the above at run time to get the OS proxy config, and pass that to requests...
...but then I found this post, which states that requests will use the OS level configuration if nothing is specified: How to use requests library without system-configured proxies
So, at this point, I can't find a definitive answer in the documentation either for requests or urllib3, but do have a SO post stating requests will use the OS level config, by calling urllib.requests.getproxies() itself.
...so can anyone confirm/deny this is the case?
thanks!
There are two aspects in your question
1. does requests use urllib.request.getproxies ?
As of version requests=2.25.1, from Session.request source, if not provided, proxy information is obtained from self.merge_environment_settings
if self.trust_env:
# Set environment's proxies.
no_proxy = proxies.get('no_proxy') if proxies is not None else None
env_proxies = get_environ_proxies(url, no_proxy=no_proxy)
And get_environ_proxies uses getproxies that is either imported from urllib (py2) or from urllib.request (py3).
So the answer is YES
2. is urllib.request.getproxies able to pick up the OS proxy configuration on windows ?
As far as I know, "the OS configured one" is not reliable on windows. At least on my corporate machine, urllib.request.getproxies does not pick up the proxy. From its documentation or from the one in ProxyHandler it states
If no proxy environment variables are set, then in a Windows environment proxy settings are obtained from the registry’s Internet Settings section, and in a Mac OS X environment proxy information is retrieved from the OS X System Configuration Framework.
From the source code I see that it reads under HKEY_CURRENT_USER > 'Software\Microsoft\Windows\CurrentVersion\Internet Settings', the value of ProxyEnable and ProxyServer. On my machine, that has a proxy configured, this is empty - the settings seem to be rather stored in Internet Explorer / the .Net stack somewhere.
Note that very often in corporate environments the proxy is set from a .pac :
So to conclude on windows at least as of today, we can not reliably trust urllib.request.getproxies. This is why I developed envswitch to make it extremely easy for me and my colleagues to switch all the proxy-related environment variables in one click, back and forth (home-train-plane/office). At least urllib (and requests) use them reliably when they are set. (note: the tool works fine even if there is a "build failed" badge on the tool's doc page :) )
I'm able to get an unsecured FTP Client/Server system going, but when I try throwing in the SSL io handlers, setting up both apps to use sslvTLSv1, it shows Connected for the Client status then eventually times out (the only Server message I get is Socket Error # 10060).
After many trials and tribulations trying to resolve this issue, I've determined that there are serious problems with enabling a certificate-less security system; meaning that, if you want it secured (with the current Indy code), you need to use certificates. Perhaps there are some settings in the SSL component that need to be made, but there just isn't specific enough info (working examples of certificate-less SSL) to make this work. Hopefully this deadlock will be resolved in a future release of Indy ;)
I have a classic ASP website running on a Windows Server 2012 box. One page makes a HTTP request to another application over https using code like this:
Sub ShopXML4http(url, inStr, outStr, method, xmlerror)
Dim objhttp
Set objhttp = Server.CreateObject ("MSXML2.ServerXMLHTTP.6.0")
objHttp.open method, url, false
If Method="POST" Then
objHttp.Send instr
Else
objHttp.Send
End if
outstr=objHttp.responseText
Set objhttp=nothing
End Sub
This code works fine almost all of the time (thousands of requests per day), but sporadically it will fail with a message like this:
Number: -2147012739
Description: An error occurred in the secure channel support
Source: msxml6.dll
The application was recently moved from an old Windows 2003 Server to the 2012 Server, and this issue never seemed to be a problem on the old server. In addition, while this error is happening on the website, I could run the exact same code in a VBScript and it works fine. Resetting the application pool seems to cause the site to be able to do the secure HTTP requests again (although it often fixes itself before I can get to the server).
I have had the exact same problem after migrating from 2003 to 2008 R2 and found the solution. Change:
Set objhttp = Server.CreateObject ("MSXML2.ServerXMLHTTP.6.0")
to:
Set objhttp = Server.CreateObject ("MSXML2.XMLHTTP.6.0")
and your problem will go away.
I tried to find the pros and cons about both objects, but haven't yet found a reason to not use XMLHTTP.
I've had the same issue and tried lots of solutions offered under a variety of posts but ultimately had no success, until now. I'll detail the solution that worked for me with reference to the problem as in my case it was PayPal. I've not opened a new post as this might not be just a paypal issue in future.
The solution is a combination of a number of stackoverflow posted solutions to similar problems but this seemed the best one to add to.
The problem
Trying to test PayPal IPN on Windows Server 2008 using classic ASP using the PayPal Sandbox returns the error "An error occurred in the secure channel support".
Why it is a problem
PayPal is requiring all communications with their systems to be as secure as possible. You will need a connection that is TLS 1.2. Windows Server 2008 is not TLS 1.2 by default.
PayPal threw some confusion into the mix by saying you need a Verisign G5 certificate, which you do for the server root but not the domain you are running your code on. I also didn't install any PayPal certificates as I don't use the API. I don't believe you need your comms from an HTTPS site either - although my domain is secured using a standard GoDaddy EV cert although I did a test on a non HTTPS site after and that worked too.
My solution
First check which kind of security your server is using via SSL Labs.
It should be TLS1.2 or higher and no other TLS's or SSL's. It must also have a SHA256 encryption.
You may need to patch the server: https://support.microsoft.com/en-us/kb/3106991.
Use IISCrypto to set the correct TLS and ciphers. I used the registry changes offered up elsewhere on stackoverflow but this did not work and actually totally screwed up my server for everything using HTTPS posts, not just my development site! IISCrypto also handles the ciphers.
Make sure your application pool is v4.5, which in itself is unclear because IIS might only offer v4.0 as an option. However this is probably actually v4.5. You can verify this via https://msdn.microsoft.com/en-us/library/hh925568(v=vs.110).aspx.
Within your code you need to use Server.CreateObject ("MSXML2.XMLHTTP.6.0"), not Server.CreateObject ("MSXML2.ServerXMLHTTP.6.0") as mentioned above.
Now I've no idea why the non-server XMLHTTP works as that seems contrary to the documentation behind it. Right now, after 10 days of stress, panic and frustration I don't care! I hope this is useful for others.
Finding the solution was a nightmare so I'll add some phrases below to help others if searching:
PayPal IPN failing with server error
PayPal SSL Windows 2008 errors
An error occurred in the secure channel support
classic ASP PayPal Sandbox SSL errors
I'd like to publicly thank Rackspace and GoDaddy for their help with this. I'd like to publicly state that I found paypal have the worst technical support ever and just do not care, constantly pointing to their own docs, if they ever respond. They say they've been sending emails out about this since September 2014 but I never received one. These new requirements are active on the PayPal Sandbox but go live in September 2016. I only came across it as developing a new solution so needed the sandbox - if you're running live you won't know about the problem until it hits and then you're dead in the water. Test your entire payment system on the PayPal sandbox asap is my advice!!
None of the answers above applies to my situation. Then I hopped on the link here:
https://support.microsoft.com/en-za/help/3140245/update-to-enable-tls-1-1-and-tls-1-2-as-a-default-secure-protocols-in
This update provides support for Transport Layer Security (TLS) 1.1 and TLS 1.2 in Windows Server 2012, Windows 7 Service Pack 1 (SP1), and Windows Server 2008 R2 SP1.
Applications and services that are written by using WinHTTP for Secure Sockets Layer (SSL) connections that use the WINHTTP_OPTION_SECURE_PROTOCOLS flag can't use TLS 1.1 or TLS 1.2 protocols. This is because the definition of this flag doesn't include these applications and services.
This update adds support for DefaultSecureProtocols registry entry that allows the system administrator to specify which SSL protocols should be used when the WINHTTP_OPTION_SECURE_PROTOCOLS flag is used.
This can allow certain applications that were built to use the WinHTTP default flag to be able to leverage the newer TLS 1.2 or TLS 1.1 protocols natively without any need for updates to the application.
This is the case for some Microsoft Office applications when they open documents from a SharePoint library or a Web Folder, IP-HTTPS tunnels for DirectAccess connectivity, and other applications by using technologies such as WebClient by using WebDav, WinRM, and others.
This update will not change the behavior of applications that are manually setting the secure protocols instead of pass the default flag.
Client service on Windows 2008 R2 server outbound to server over TLS reciprocated the error in question. I thought it could be cipher suite compatibility. Wireshark trace indicated version in Client Hello request was TLS 1.0 but server requires TLS 1.2. The cipher suites sent to outbound server from client service were fine. The problem is the client service or application on Windows server default employs the system default, which is not TLS 1.2.
The solution is to add a registry subkey named DefaultSecureProtocols with a value corresponding to which TLS version(s) should be supported. Add said registry subkey, with type DWORD, to the following locations:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp
For Internet Explorer fix, you can add a similar registry subkey titled SecureProtocols, also with type DWORD, to the following locations:
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings
Below you can find the table of values for both subkeys:
DefaultSecureProtocols Value Protocol enabled
0x00000008 Enable SSL 2.0 by default
0x00000020 Enable SSL 3.0 by default
0x00000080 Enable TLS 1.0 by default
0x00000200 Enable TLS 1.1 by default
0x00000800 Enable TLS 1.2 by default
For example:
The administrator wants to override the default values for WINHTTP_OPTION_SECURE_PROTOCOLS to specify TLS 1.1 and TLS 1.2.
Take the value for TLS 1.1 (0x00000200) and the value for TLS 1.2 (0x00000800) then add them together in calculator (in programmer mode), the resulting registry value would be 0x00000A00.
I applied 0x00000A00 as the value for both subkeys and it successfully resolved the issue.
There is also an Easy Fix (link is here: https://aka.ms/easyfix51044) available from Microsoft, if you don't wish to manually enter registry subkeys and values.
It's all valid however the 'critical' missing bit for TLS1.2 support on Windows 7 with IIS7.5 and classic asp is setting this in the registry:-
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp]
"DefaultSecureProtocols"=dword:00000800
I hope that saves you a day of faffing, rebooting and head scratching! :)
This code snippet is useful for testing. https://www.howsmyssl.com/
<%
Set winhttp = Server.CreateObject("WinHTTP.WinHTTPRequest.5.1")
winhttp.open "GET", "https://howsmyssl.com/a/check", False
winhttp.Send
Response.Write winhttp.responseText
%>
In a Windows Server 2016 Classic ASP script, fetching an HTTPS URL from Windows Server 2012 R2, I recently had to remove SSL 2.0 from SecureProtocols in order to stop this secure channel error -2147012739.
' Use the latest client
Set httpClient = Server.CreateObject("WinHttp.WinHttpRequest.5.1")
' allow only TLS 1.2 or TLS 1.1
Const WHR_SecureProtocols = 9
httpClient.Option(WHR_SecureProtocols) = &h0800 + &h0200
' Other values: TLS 1.0 &h0080, SSL 3.0 &h0020, SSL 2.0 &h0008
' NB Including SSL 2.0 stops https to Windows Server 2012 R2 working
' Other options you may want to set, from https://learn.microsoft.com/en-us/windows/desktop/winhttp/winhttprequestoption
' Ignore certificate errors
Const WHR_SslErrorIgnoreFlags = 4
httpClient.Option(WHR_SslErrorIgnoreFlags) = &h3300
' Don't bother checking cert, or risking failure if we can't check
Const WHR_EnableCertificateRevocationCheck = 18
httpClient.Option(WHR_EnableCertificateRevocationCheck) = False
Troubleshooting error codes:
-2147012739 is a HRESULT.
In hexadecimal that's 0x80072F7D.
Look at the LOWORD: 0x2F7D.
Convert that back to decimal: 12157.
Lookup 12157 error codes.
Find that it matches: ERROR_WINHTTP_SECURE_CHANNEL_ERROR
A bit of Google-fu finds http://msdn.microsoft.com/en-us/library/windows/desktop/aa383770(v=vs.85).aspx which states:
ERROR_WINHTTP_SECURE_CHANNEL_ERROR
12157
Indicates that an error occurred having to do with a secure channel (equivalent to error codes that begin with "SEC_E_" and "SEC_I_" listed in the "winerror.h" header file).
However, you already discovered this as the message you got was "Description: An error occurred in the secure channel support". So this leads us right back where we started.
The other observation I make is that your code is a non-asynchronous WinHTTP request (I know it has to be to function inside ASP), but, the concern is, due to the high frequency, your machine could be processing more than one WinHTTP request concurrently. I've seen some Windows deliberately throttle the total number of active concurrent WinHTTP request by blocking the late requests. For example, on a Windows 7 machine a process cannot make more than 2 concurrent requests to the same remote server. i.e. The 3rd, 4th... requests will be blocked until the first two complete.
One solution is to load balance incoming request over more than one application pool or over more servers.
We had a variation on this issues and it really cost us some time to figure it out.
Here is the situation: An older Linux server hosting an application written in PHP and provides data through webservice calls. The server is using HTTPS. Calls from various clients are made with code using the winHTTP 5.2 library. (Winhttp.dll)
Symptom: Our clients are now getting sporadic error messages when making repeated winHTTP calls using a ‘POST’ command. The messages are either ‘The buffers supplied to a function was to small.‘ or ‘An error occurred in the secure channel support ‘. After much searching we discovered that the client’s server was logging ‘Schannel Event ID 36887 alert code 20’ in the Event Viewer that corresponded with the visible error message.
Solution: We discovered that our old Linux server could not support TLS 1.2. (CentOS 5.11) We also learned that several of our clients had recently (summer 2016) applied an update to their Microsoft servers. (Server 2008, server 2012) The fix was to force their servers to use TLS 1.1 for the webservice calls. The part that is rather strange to me is that the settings in Internet Explorer for changing the TLS had no effect on the problem. However by changing a setting in Group Policies we were able to solve the problem. Our technical advisor on this matter pointed out that the change is really obscure, but that a third-party vendor has provided a quick solution. That tool is called IIS Crypto from Nartac. https://www.nartac.com/Products/IISCrypto/Download
The tool lets you specifically select Protocols.
We are now getting a new server to host our applications (CentOS 6) and then should be able to use the TLS 1.2 protocol!
I encountered this error a few months ago myself. Most often, this issue is caused by an invalid SSL cert. Considering that at the time of the post you had just migrated to a new server, you probably just need to reinstall the SSL certificate.
I realize this question is old, but hopefully someone else can benefit from my answer.
Due to a couple of issues with my host, I'm unable to use a SSL-certificate on my server (I'm not ready to change provider just yet), and can't therefore use HTTPS. This server will communicate with a couple of client-computers and will transfer data that's somewhat secret.
Would it be reasonable to simply use AES encryption (encryption on client before sending, decryption on server before processing) instead of HTTPS?
This depends on your deployment environment.
Replacing SSL/TLS (and HTTPS) with your own encryption protocol for use by a web browser is always a bad idea, since it relies on JavaScript code delivered insecurely (for details, see this question on Security.SE, for example).
If the client isn't a web browser, you have more options available. In particular, you can implement message-level security instead of transport-level security (which is what HTTPS uses).
There are a number of attempts to standardise message-level security with HTTP. For example:
HTTPsec had a public specification (still available on WebArchive), but a commercial implementation. I'm not sure whether this has been widely reviewed.
WS-Security, oriented towards the world of SOAP.
Perhaps more simply, if you want to re-use existing tools, you could use S/MIME or PGP (in the same way as you would for e-mails) to encrypt the HTTP message entities. Unlike HTTPS, this won't protect the URL or the HTTP headers, but this might be enough if you don't put any sensitive data there.
The further down you go with "raw encryption" yourself (using AES directly, for example), the more likely you'll have to implement other aspects of security manually (typically, verifying the remote party's identity and dealing with the problem of pre-sharing the keys).
If you have a small list of clients that don't change often, you could implement your own SSL-Tunnel using SSH. On the clients do a;
ssh -D 4444 nulluser#example.com -N
where nulluser has no shell or file access on example.com.
Then add a foxyproxy whitelist setting - so that for example.com the client browsers use the localhost:4040 proxy.
It's a hack, it's totally unscalable, but it would work as I say for a small, static number of clients, and it has the advantage of not reinventing any wheels while being totally secure.
Sorry if it is a duplicate, as I am not a security nor network expert I may have missed the correct lingo to find information.
I am working on an application to intercept and modify HTTP requests and responses between a web browser and a web server (see how to intercept and modify HTTP responses on server side? for the background). I decided to implement a reverse proxy in ASP.Net which forwards client requests to the back-end HTTP server, translates links and headers from the response to the properly "proxified" URL, and sends the response to the client after having extracted relevant information from the response.
It is working as expected, except for the authentication part: the web server uses NTLM authentication by default, and just forwarding requests and responses through the reverse proxy does not allow the user to be authenticated on the remote application. Both the reverse proxy and the web application are on the same physical machine and are executed in the same IIS server (Windows server 2008/IIS 7 if that matters). I tried both enabling and disabling authentication on the reverse proxy app with no luck.
I have looked for information about it, and it seems to be related to the "double-hop problem", which I do not understand. My question is: is there a way to authenticate the user on the remote application through the reverse proxy using NTLM? If there is none, are there alternative authentication methods I could use?
Even if you don't have a solution to my problem, just pointing me to relevant information about it to help me get out of the confusion would be great!
I found what the problem was (and it is NTLM): in order to have the browser asks the user for its credentials, the response must have a 401 status code. My reverse proxy was forwarding the response to the browser, so IIS was adding a standard HTML code to explain the requested page cannot be accessed thus preventing the browser from asking credentials.
The problem was solved by removing the response content when the status code is a 401.
With all due respect I have for the one that answered that some years ago, I must admit this is plainly false. The problem was indeed solved AFTER removing the response content when the status code is a 401, but it had none to do with the initial problem..
The truth is that windows authentication was made to authenticate people over local windows networks, where no proxy server is present or even needed.
The main problem with NTLM authentication is that this protocol does not authenticate the HTTP session but the underlying TCP connection, and as far as I know there is no way to access it from asp code.
Every proxy server I tried broke NTLM authentication.
Windows authentication is comfortable for an user because he won't ever need to enter your password to whatever application may lie in your intranet, frightening for a security guy because there is an auto-login without even a prompt if the site domain is trusted by IE, shocking for a network administrator because it melts the application, transport and network layer into some "windows ball of mug" instead of just plain http traffic.
NTLM won't work if the TCP packets are not forwarded exactly as the reverse proxy received > them. And that's why many reverse proxy doesn't work with NTLM authentication. (like nginx) > They forward HTTP requests correcty but not the TCP packets.
Nginx has the functionality to work with NTLM authentication. Keepalive needs to be enabled which is only available trough the http_upstream_module. Additionally in the location block you need to specify that you will be using HTTP/1.1 and that the "Connection" header field should be cleared for each proxied request. Nginx config should look something like:
upstream http_backend {
server 1.1.1.1:80;
keepalive 16;
}
server {
...
location / {
proxy_pass http://http_backend/;
proxy_http_version 1.1;
proxy_set_header Connection "";
...
}
}
I scratched my head for quite some time with this issue but the above works for me. Note that if you need to proxy HTTPS traffic, a separate upstream block is deemed necessary. To clarify a bit more, "keepalive 16;" specifies the number of simultaneous connections to the upstream your proxy is allowed to keep. Adjust the number as per the expected number of simultaneous visitors on the site.
Although this is an old post, I just want to report that it works for me quite well with an Apache2.2 reverse proxy and the keepalive=on option. Obviously, this keeps the connection between the proxy and the SharePoint host open and "pinned" to the client<>proxy connection. I don't exactly know the mechanisms behind this, but it works fairly well.
But: Sometimes, my users encounter the issue that they're logged in as another user. So there seems to be some mixing-up through sessions. I will have to give this some further testing.
Solution for everything (in case you have a valid, signed SSL certificate): Switch IIS to Basic Auth. This works absolutely fine, and even Windows (i.e. Office with SharePoint connection, all WebClient-based processes etc.) won't complain at all.
But they will when you're just using http without SSL/TLS, and also with self-signed certificates.
I confirm that it works with "keep-alive=on" on apache2.2
I examined frames with Wireshark, and I know why it doesn't work. NTLM won't work if the TCP packets are not forwarded exactly as the reverse proxy received them. That's why many reverse proxies, like nginx, don't work with NTLM authentication. Reverse proxies forward HTTP requests correctly but not the TCP packets.
NTLM requires a TCP reverse proxy.