I am using NGINX to offload SSL and then using varnish to load balance/cache responses from IIS8.
i.e. ssl -> nginx -> varnish -> iis8
This works perfectly. SignalR using long polling works fine - all my timeouts etc are properly configured.
Now I'm trying to get SignalR and websockets to work with this config; without success. To simplify things I've tried the following two set ups:
nginx -> iis8 (with these settings)
varnish -> iis8 (with these settings)
Neither of these work. I keep getting this error in Chrome:
WebSocket connection to 'ws://xxxxxxxx' failed: Unexpected response code: 500
and this from IIS:
System.InvalidOperationException: Not a valid web socket request.
at Microsoft.AspNet.SignalR.Owin.ServerRequest.AcceptWebSocketRequest(Func`2 callback)
at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(HostContext context)
at Microsoft.AspNet.SignalR.Owin.CallHandler.Invoke(IDictionary`2 environment)
at Microsoft.Owin.Host.SystemWeb.OwinCallContext.Execute()
at Microsoft.Owin.Host.SystemWeb.OwinHttpHandler.BeginProcessRequest(HttpContextBasehttpContext, AsyncCallback callback, Object extraData)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Owin.Host.SystemWeb.CallContextAsyncResult.End(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
It appears that something is not being correctly forwarded to IIS or IIS is being picky about something. Can anyone point me towards a working config for this set up?
Related
We are running a Service Fabric application on our remote dev cluster. It consists of several stateful and stateless services and is fronted by several front-end APIs running on Kestrel.
Until now, since it was not used for production, Kestrel was configured to use a self-signed certificate, which was also used for the reverse proxy and the cluster itself and the service was running directly on the default domain provided by Azure, <app>.<region>.cloudapp.azure.com.
We are now getting to the point in development where the self-signed certificate errors are becoming problematic, with third party callbacks rejecting the connection, so it was seen as the time to start using a proper domain and certificate for it.
So far, I have done the following:
Added an A record for devcluster.somexampledomain.com -> our public IP for the service.
Created a Wildcard Azure Application certificate for *.someexampledomain.com.
Imported the certificate to Azure Key Vault.
Bound the certificate to the Vault Secrets of the cluster, pulling the certificate to Cert:/LocalMachine/My/
Modified the application config to use this certificate when initialising Kestrel and verified that it is found when it is initialising.
Have tried with and without UseHsts() and UseHttpsRedirection()
Kestrel is configured with Listen(IPAddress.IPv6Any, endpoint.Port, ...) and UseHttps(X509Certificate2) on the options object.
UseUrls(string) is used with the default Url, which is https://+:<port> but tried manually adding https://*:<port> and even the actual hostname itself.
No matter what I have tried, no HTTPS connection can be established to the server. Trying the endpoints of the other staging servers that still use the old certificate, it works as expected.
Using openssl s_client -connect devcluster.someexampledomain.com:<port> -prexit, I get:
---
no peer certificate available
---
No client certificate CA names sent
---
There are no errors or exceptions being logged on ETW, everything seems to be in order. I suspect that this might have something to do with the CN of the certificate but I have run out of ideas to try and find out what is going on and how to fix it.
Been trying to look into this using Fiddler and I am not getting much out of it, the session just ends with fiddler.network.https> HTTPS handshake to <myhost> (for #191) failed. System.IO.IOException Authentication failed because the remote party has closed the transport stream.
Does anybody know how to add some logging on the Kestrel side? I don't think installing Fiddler on the Azure VMs running my cluster is a viable solution.
After delving into the Kestrel source, I found that it logs under "Microsoft-AspNetCore-Server-Kestrel" and "Microsoft-Extensions-Logging", so adding transfer of those I found what was happening.
Connections were terminating with the following exception:
System.ComponentModel.Win32Exception (0x8009030D): The credentials supplied to the package were not recognized
at System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface secModule, String package, CredentialUse intent, SCHANNEL_CRED scc)
at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(CredentialUse credUsage, SCHANNEL_CRED secureCredential)
at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(X509Certificate certificate, SslProtocols protocols, EncryptionPolicy policy, Boolean isServer)
at System.Net.Security.SecureChannel.AcquireServerCredentials(Byte[]& thumbPrint, Byte[] clientHello)
at System.Net.Security.SecureChannel.GenerateToken(Byte[] input, Int32 offset, Int32 count, Byte[]& output)
at System.Net.Security.SecureChannel.NextMessage(Byte[] incoming, Int32 offset, Int32 count)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionAdapter.InnerOnConnectionAsync(ConnectionAdapterContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ApplyConnectionAdaptersAsync()
This makes it a manifestation of Certificate problem with a new machine - credentials supplied to package not recognized.
I spent some time trying to figure out what the best way to sort this out would be, the Service Fabric documentation has a script to modify the permissions but that just did not sound right.
As it turns out, this can be done directly in the ApplicationManifest as follows:
<Principals>
<Users>
<User Name="NETWORK SERVICE" AccountType="NetworkService" />
</Users>
</Principals>
<Policies>
<SecurityAccessPolicies>
<SecurityAccessPolicy ResourceRef="HttpsCert2" PrincipalRef="NETWORK SERVICE" ResourceType="Certificate" />
</SecurityAccessPolicies>
</Policies>
<Certificates>
<SecretsCertificate X509FindValue="[HttpsCertThumbprint]" Name="HttpsCert" />
</Certificates>
For the SecurityAccessPolicy to find the ResourceRef it had to be a SecretsCertificate, not an EndpointCertificate. Since the EndpointBindingPolicy requires an EndpointCertificate, I just added both a SecretsCertificate and an EndpointCertificate, with different names. They are both referring to the same certificate, so it worked. It doesn't feel particularly clean having to double them up but that is the solution I have for now.
After a deploy to our Azure Web App, we are getting 500 timeouts on any request to the service:
500 - The request timed out.
The web server failed to respond within the specified time.
This has come out of the blue and we cannot determine what's causing it. It seems to take around 230s consistently to time out.
I've enabled all the diagnostic logs in the portal:
But I honestly don't quite know what to look for in the logs. I've scoured through all the files in the following folders but nothing jumps out.
How can I troubleshoot this problem?
The trick to get debug messages is to set stdoutLogFile="D:\home\LogFiles\stdout.log" in your config, instead of the ..\logs path that you had. After changing that, you get an error file under D:\home\LogFiles. Here is the error you get:
Application startup exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentNullException: SMTP server password cannot be null or empty.
Parameter name: smtpPassword
at TransitApi.Infrastructure.Modules.Logging.EmailOutput.EmailLogger..ctor(String recipient, String sender, String smtpUsername, String smtpPassword, String smtpHost, Int32 smtpPort, String environmentName, LogLevel minimumLevel)
at TransitApi.Infrastructure.Modules.Logging.EmailOutput.EmailLoggerProvider.CreateLogger(String name)
at Microsoft.Extensions.Logging.Logger.AddProvider(ILoggerProvider provider)
at Microsoft.Extensions.Logging.LoggerFactory.AddProvider(ILoggerProvider provider)
at TransitApi.Api.Startup.Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
So some kind of issue setting up the mail server. That causes the process to crash, and that it behaves poorly.
But I highly suggest that you upgrade to Core RC2, as RC1 is quite obsolete.
Asp.net mvc 5 application web config file is
sessionState mode="Custom" customProvider="RedisSessionProvider">
providers>
add name="RedisSessionProvider" type="Microsoft.Web.Redis.RedisSessionStateProvider" port="6380" host="XXX.redis.cache.windows.net" accessKey="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" ssl="true" />
providers>
sessionState>
When i run application have eror
Server Error in '/' Application.
No connection is available to service this operation: EVAL
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: EVAL
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[RedisConnectionException: No connection is available to service this operation: EVAL]
StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl(Message message, ResultProcessor`1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:1922
StackExchange.Redis.RedisBase.ExecuteSync(Message message, ResultProcessor`1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:80
StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:866
Microsoft.Web.Redis.<>c__DisplayClass4.<Eval>b__3() in c:\TeamCity\buildAgent\work\f55792526e6d9089\src\Shared\StackExchangeClientConnection.cs:113
Microsoft.Web.Redis.StackExchangeClientConnection.RetryForScriptNotFound(Func`1 redisOperation) in c:\TeamCity\buildAgent\work\f55792526e6d9089\src\Shared\StackExchangeClientConnection.cs:129
Microsoft.Web.Redis.StackExchangeClientConnection.RetryLogic(Func`1 redisOperation) in c:\TeamCity\buildAgent\work\f55792526e6d9089\src\Shared\StackExchangeClientConnection.cs:155
Microsoft.Web.Redis.StackExchangeClientConnection.Eval(String script, String[] keyArgs, Object[] valueArgs) in c:\TeamCity\buildAgent\work\f55792526e6d9089\src\Shared\StackExchangeClientConnection.cs:113
Microsoft.Web.Redis.RedisConnectionWrapper.Set(ISessionStateItemCollection data, Int32 sessionTimeout) in c:\TeamCity\buildAgent\work\f55792526e6d9089\src\RedisSessionStateProvider\RedisConnectionWrapper.cs:135
Microsoft.Web.Redis.RedisSessionStateProvider.SetAndReleaseItemExclusive(HttpContext context, String id, SessionStateStoreData item, Object lockId, Boolean newItem) in c:\TeamCity\buildAgent\work\f55792526e6d9089\src\RedisSessionStateProvider\RedisSessionStateProvider.cs:408
System.Web.SessionState.SessionStateModule.OnReleaseState(Object source, EventArgs eventArgs) +906
System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +142
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +92
Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.6.1055.0
I had the same, and eventually fixed it by adding sslprotocols=tls12 to the connectionstring. It should not be necessary, but apparently it is, using recently created Redis caches on Azure. Probably due to the deprecation of TLS 1.0 and 1.2
Could be for 3 reasons:
Your connection string is wrong (looks ok any way)
The instance isn't yet created (usually takes a lot to be created in azure)
You are facing firewall (including windows firewall) issues
Are you able to get connected to the redis cache instance using command prompt?
Download redis for windows Here
and then run
redis-cli -h XXX.redis.cache.windows.net -a XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
You can find here a detailed demo (in spanish but console demos remains usable.)
NinjaTip #18 - Introducción a Redis
For testing purposes ensure than Redis service has port 6379 open (this is the non SSl port option you need to enable on azure)
If you can't connect to your instance is because some of the reasons listed above.
I had the same issue before and I found a solution that using another one:
https://github.com/alex-simonov/RedisAspNetProviders
This one works for me
I found that only the following configuration in web config works for me:
<sessionState mode="Custom" customProvider="MySessionStateStore">
<providers>
<add name="MySessionStateStore" type="Microsoft.Web.Redis.RedisSessionStateProvider" connectionString="PUTCORRECTNAMEHERE.edis.cache.windows.net:6380,password=YOURPRIMARYKEYPUTHERE,ssl=True,abortConnect=False" />
</providers>
</sessionState>
I've got an ASP.NET 4.5 web app using SignalR 1.0.0-rc1 which I've pushed out to an Azure website for some quick-and-dirty testing (I'm mostly interested in this page here: http://alantaappbeta.azurewebsites.net/api/v3.0/Tests/Sample.htm).
The problem is that the page doesn't seem to want to talk to the SignalR service. The hubs and everything are registered correctly, as http://alantaappbeta.azurewebsites.net/signalr/hubs returns the correct client-side hubs file, and the call to /signalr/negotiate returns some reasonable looking JSON.
{
"Url":"/signalr",
"ConnectionId":"a15023f9-c675-4fc2-9fd6-403a297f10c0",
"KeepAlive":15.0,
"DisconnectTimeout":40.0,
"TryWebSockets":false,
"WebSocketServerUrl":null,
"ProtocolVersion":"1.1"
}
But when it calls /signalr/ping, it returns a 500 error, with the message "Protocol error: Unknown transport". The stack trace returned in the error page looks like this:
[InvalidOperationException]: Protocol error: Unknown transport.
at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequestAsync(HostContext context)
at Microsoft.AspNet.SignalR.Hubs.HubDispatcher.ProcessRequestAsync(HostContext context)
at Microsoft.AspNet.SignalR.Owin.CallHandler.Invoke(IDictionary`2 environment)
at Microsoft.AspNet.SignalR.Owin.Handlers.HubDispatcherHandler.Invoke(IDictionary`2 environment)
at Microsoft.Owin.Host.SystemWeb.OwinCallContext.Execute()
at Microsoft.Owin.Host.SystemWeb.OwinHttpHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object extraData)
at Microsoft.Owin.Host.SystemWeb.OwinHttpHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Because this is on Azure, I don't have access to all the troubleshooting tools you'd normally have, but what logs I have been able to get access to don't give me any information beyond the above.
I suppose it goes without saying that this runs fine on my local IIS instance :-).
Any suggestions?
Edit: This is how I'm opening my connection:
$.connection.hub.start({
transport: 'auto',
xdomain: true
}).done(function () {
console.log('Connected with hub.id=' + $.connection.hub.id);
}).fail(function (e) {
console.log('Unable to connect to SignalR Hubs: ' + e);
});
But neither the .done() nor the .fail() handlers are being called.
Oddly enough, if I set the transport to 'longPolling', I can get it to fail in a similar manner using IIS Express on my local box. But it doesn't seem to make any difference what I set the transport to on Azure: I still get the same error.
It turns out that the issue was that I had a mix of different versions of SignalR on my machine: I was using the server-side DLLs from the NuGet package (1.0.0-rc1), but the JavaScript client file was from the dev branch. Once I got them all synchronized, everything worked. I'm still not sure precisely why the issue only showed up with Azure, but I suspect it has something to do with the precise transports supported by the different versions of IIS.
We’re getting the following error message when we click on “Search Settings” for a Shared Services Provider: “Authentication failed because the remote party has closed the transport stream.”
This is a new server environment with two web front ends, one database server, and one index server, all running Windows 2003 x64.
Does anyone have any thoughts related to if this could be related to 64-bit, or what could be causing the error.
Here are the full details from ULS:
09/17/2008 16:30:34.13 w3wp.exe (0x0E84) 0x030C Search Server Common MS Search Administration 86x4 High Configuring the Search Application web service Url to 'https://mushni-sptwb04q:56738/Shared%20Services%20Portal/Search/SearchAdmin.asmx'.
09/17/2008 16:30:34.14 w3wp.exe (0x0E84) 0x030C Search Server Common MS Search Administration 86ze High Exception caught in Search Admin web-service proxy (client). System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream. at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult) at System.Threading.ExecutionContext.runTryCode(Object userData) at System.Runtime.Co...
09/17/2008 16:30:34.14* w3wp.exe (0x0E84) 0x030C Search Server Common MS Search Administration 86ze High ...mpilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result) at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.Net.ConnectStream.WriteHeaders(Boolean async) --- End of inner exception stack trace --- at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request) at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request) at System.Web.Services.Protocols.SoapHt...
09/17/2008 16:30:34.14* w3wp.exe (0x0E84) 0x030C Search Server Common MS Search Administration 86ze High ...tpClientProtocol.Invoke(String methodName, Object[] parameters) at Microsoft.Office.Server.Search.Administration.SearchWebServiceProxy.RunWithSoapExceptionHandling[T](String methodName, Object[] parameters)
I guess you find this exception in the index server, right?
Are you able to browse to 'https://mushni-sptwb04q:56738/Shared%20Services%20Portal/Search/SearchAdmin.asmx' from the index server?
It seems like SSL is not properly provisioned on the front-end servers. This might solve your issue:
Remove the SSL certificate of the front-end servers
Remove the index server from the farm
Move the search and index roles to one of the front-ends
Join the index server back to the farm
Add the index/search roles to the index server
Apply the SSL certificate (you can generate them using SelfSSL) to both front-ends
Be careful with SelfSSL, its better to use Use SSLDiag. SelfSSL has a bug where if you use it to assign certificates to multiple sites on the same box, only the last site will work. You can run SslDiag from the command line like so:
ssldiag /selfssl /V:999 /N:CN=<hostname> /S:<siteId>
Use metabase explorer to find the side it.
Could be an SSL issue.
Do have a look into profiles settings, do you get any error when accessing to the User Profiles settings for that same SSP?
I'm having the same problem. The "Office Server Web Services" (henceforth OSWS) site is available through HTTP on my app server, but not via HTTPS. It doesn't matter where I try to hit the HTTPS URL from, it just flat-out fails (read: no HTTP error code).
However, I have come up with some more information. When the app server was joined to the farm, it gave OSWS a different site identifier than exists in the rest of the farm.
I tried changing the site identifier, but that didn't work. I've also tried installing the IIS diagnostics toolkit. That pointed me towards the certificate that MOSS installed when the machine was joined to the farm. The line of interest is this one:
#WARNING: AcquireCredentialsHandle failed with error -2146893043(0x8009030d)
Unfortunately, it looks like Microsoft has embedded some information in the certificate that would prevent me from using SelfSSL or similar tools. Here's the subject (suitably scrubbed):
CN={hostname},L=951338967,OU=SharePoint,O=Microsoft
The "L" parameter matches the original (and incorrect) site identifier that the site was given and not the one that matches the rest of the farm.
My next step is to see if I can generate something that looks appropriate and install it with winhttpcertcfg.exe
We are also running x64 windows and moss 2007 with .net 3.5 sp1,same issues. I suspect this is the culprit.
To resolve this issue download the IIS6 resource kit and run the following command
Selfssl /s:(IIS ID of the Office Server Web Services site) /v:9999
Cheers,
-Ivan