My website uses ASP.NET 5 with RC1-Update1 and some communication with the client goes through web sockets (SignalR is used).
During testing I found out that while running on IIS (or IIS Express), communication doesn't work through sockets. SignalR fallbacks to long polling.
SignalR log:
SignalR: Client subscribed to hub 'chathub'.
SignalR: Negotiating with '/signalr/signalr/negotiate?clientProtocol=1.5&connectionData=%5B%7B%22name%22%3A%22chathub%22%7D%5D'.
SignalR: serverSentEvents transport starting.
SignalR: Attempting to connect to SSE endpoint 'http://localhost:31650/signalr/signalr/connect?transport=serverSentEvents&c…wVlnXCY8bI7R8E&connectionData=%5B%7B%22name%22%3A%22chathub%22%7D%5D&tid=2'.
SignalR: serverSentEvents transport timed out when trying to connect.
SignalR: EventSource calling close().
SignalR: serverSentEvents transport failed to connect. Attempting to fall back.
SignalR: foreverFrame transport starting.
SignalR: Forever Frame is not supported by SignalR on browsers with SSE support.
SignalR: foreverFrame transport failed to connect. Attempting to fall back.
SignalR: longPolling transport starting.
SignalR: Opening long polling request to 'http://localhost:31650/signalr/signalr/connect?transport=longPolling&client…8WRrhWwVlnXCY8bI7R8E&connectionData=%5B%7B%22name%22%3A%22chathub%22%7D%5D'.
SignalR: Long poll complete
After long hours of debugging, I started my website directly through dnx (no IIS or anything else) via command line using:
dnx web
And web sockets worked out of the box without any problems.
During my research towards the solution I found out that in nodejs with socket.io web apps, web sockets in IIS must be disabled so that they are correctly forwarded to nodejs (link to resource here on iis.net), so I tried the same thing and got the error (occurs as soon as webSocket element is present in web.config):
An error occurred attempting to determine the process id of the DNX process hosting your application.
My web.config:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<handlers>
<add name="httpPlatformHandler" path="*" verb="*" modules="httpPlatformHandler" resourceType="Unspecified" />
</handlers>
<httpPlatform processPath="%DNX_PATH%" arguments="%DNX_ARGS%" forwardWindowsAuthToken="false" stdoutLogEnabled="true" startupTimeLimit="3600" />
<staticContent>
<!-- A bunch of mime types (no difference if removed) -->
</staticContent>
<webSocket enabled="false" />
</system.webServer>
</configuration>
At this point I ran out of ideas. I made sure I have WebSockets installed in windows features and tried deploying my website on Azure (with web sockets turned on) where it will be hosted when finished.
I'd say that the problem is web sockets not being forwarded to DNX, but cannot find the solution for it. Any help would be greatly appreciated.
Update
Solution where the problem is recreated.
The problem was in missing package Microsoft.AspNet.WebSockets.Server and call in startup.cs app.UseWebSockets().
The issues was resolved on GitHub, big thanks to #davidfowl.
Related
After using deploying my .NET Core 2.1 App to my server, I get the following error when I access the page:
HTTP Error 502.5 - Process Failure
Following Microsoft's always helpful link on the same page (cough cough), I checked the Event View Log and it states:
Application 'MACHINE/WEBROOT/APPHOST/BLAH.COM' with physical root
'D:\inetpub\vhosts\BLAH.com\httpdocs\' failed to start process with
commandline 'dotnet .\blah.dll', ErrorCode = '0x80070002 : 0.
The message was not only unhelpful, but completely irrelevant. I tried to enable the web app's logging in web.config:
<aspNetCore requestTimeout="23:00:00" processPath="dotnet" arguments=".\blah.dll" forwardWindowsAuthToken="false" stdoutLogEnabled="true" stdoutLogFile=".\logs\stdout" startupTimeLimit="3600" />
Oddly, no files were created in the Log directory until I granted permission to my IWPD process. The files it created weren't in the stdout directory as specified, but were in the parent log directory. They also were empty.
I ran the web app from the command line using:
dotnet .\blah.dll
The application appeared to run and I could see some messages on the screen, but I still got the same 502.2 error accessing the site with no additional information in the logs. I followed the troubleshooting steps for ASP.NET Core 2.1 and it mentioned this:
If the errors occur when making a request to the app, make a request
to the host and port where Kestrel listens. Using the default host and
post, make a request to http://localhost:5000/. If the app responds
normally at the Kestrel endpoint address, the problem is more likely
related to the reverse proxy configuration and less likely within the
app.
I ran the web app from the command line and tried to access it using http://localhost:5000. I was finally able to access the site, but I still got the same 502.2 error via IIS.
It's obvious IIS and Kestrel are not communicating with each other. I don't understand why there's two web servers and how to get them to communicate with each other.
Solved the problem. The installer for the server's IIS module doesn't restart or warn the user to restart IIS. Restarting IIS loaded the necessary module. Clearly, this is a bug by any other name.
I am facing some problem with socket getting keep polled continuously without gap. Not sure whats happening.
If same code placed in digital ocean cloud, it's not happening.
Little background:
I'm running azure VM behind load balancers.
If you created load balancer using Application Request Routing with IIS, you'd need to Disable WebSocket when using socket.io on node.js.
If you are using the WebSocket support in socket.io on node.js in your
site, you will need to disable the default IIS WebSockets module by
adding the below snippet to your web.config or applicationHost.config.
If this is not done, the IIS WebSockets module will attempt to handle
the WebSocket communication rather than letting this task fall through
to node.js (and hence your application). This will result in
unexpected errors when you attempt to access your site.
<system.webServer>
...
<webSocket enabled="false"/>
</system.webServer>
Updating to latest socket library in nodeJS fixed this issue. Try your luck!
I'm hosting a small node.js app in azure, but when a client is joined it gets reconnected almost immediately and this is keep going on!
If i switch "Web Sockets" on in Azure, the reconnections are gone, but it doesn't seem to recieve any disconnect event if i close the clients*, also the connection events are registered relatively slow as well!
*disconnect events do get registered after a minute delay!
If i run the app in local, everything works fine!
You didn't share any code or web.config file with us. However, there is an official instruction: Create a Node.js chat application with Socket.IO in Azure App Service we can follow.
You may need to pay attention to Verify web.config settings:
Azure web apps that host Node.js applications use the web.config
file to route incoming requests to the Node.js application. For
WebSockets to function correctly with Node.js applications, the
web.config must contain the following entry.
<webSocket enabled="false"/>
This disables the IIS WebSockets module, which includes its own
implementation of WebSockets and conflicts with Node.js specific
WebSocket modules such as Socket.IO. If this line is not present, or
is set to true, this may be the reason that the WebSocket transport
is not working for your application.
We have a Delphi SOAP service which needs to be SSL-enabled. I opted to use an IIS ARR reverse proxy to do SSL offloading for ease of configuration (compared to OpenSSL and manual certificate + passphrase management). ARR works, but it adds an insane amount of overhead... Response time went from under 2 seconds to 19 seconds for 18 service requests (about 60Kb compressed total).
I added timestamp logging to client & server for when messages are both sent and received. It shows about 1 second added to each request routing through ARR between sending from the client and receipt by the service. The response is routed back very quickly, only the request routing via ARR is slow (see image below).
How can I trace the source of overhead? Is ARR not suited to this use case? I tried tweaking and disabling most settings, including caching. I tried different hosts with clean IIS setups, including a production Windows Server 2012. SSL itself is not the overhead, just having an ARR HTTP reverse proxy causes the delay.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<rewrite>
<rules>
<rule name="ReverseProxyInboundRule1" stopProcessing="true">
<match url="(.*)" />
<action type="Rewrite" url="http://localhost:8987/{R:1}" />
</rule>
</rules>
</rewrite>
</system.webServer>
</configuration>
Request & response samples from Fiddler:
Raw HTTP Request
Raw HTTP Response
We have the same problem. I found the root, it's in System.Net.Sockets.Socket.DoConnect
The problem is related to IPv6:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/203b6230-e4c0-477c-9a0a-0c21a7ad1615/strange-onesecond-delay-with-tcpconnections-to-localhost?forum=clr
http://msdn.microsoft.com/en-us/library/115ytk56.aspx
"If IPv6 is enabled and the TcpClient(String, Int32) method is called to connect to a host that resolves to both IPv6 and IPv4 addresses, the connection to the IPv6 address will be attempted first before the IPv4 address. This may have the effect of delaying the time to establish the connection if the host is not listening on the IPv6 address."
To resolve it for loopback requests you need to disable IPv6 on a machine, see p.4-5-6:
https://stackoverflow.com/a/12403731
My advice: use IIS for the apps, use Apache HTTP Daemon for the proxying.
In the past I've used various pieces of software and hardware for SSL offloading (starting somewhere in 2003 I think). Each with their own pricing level and features. The last years I've switched to solely using Apache HTTP Daemon for that purpose. Even in combination with IIS and on Windows. Apache is easy to configure once you have a running sample and more easily grows to more complex scenarios with forwarding and renaming.
Some instructions how to use Apache HTTP Daemon on Windows as SSL offloading engine can be found on http://www.invantive.com/about-invantive/news/entryid/897/ssl-offloading-for-apache-tomcat.
Turning off IPv6, as Dmitry suggested, solved this for me.
You could also use 127.0.0.1 in your rewrite, instead of localhost, to force IPv4.
I would say you have something screwy in your setup. We currently run ARR for SSL offloading and conducted throughput testing at significant volume and ARR showed little to no affect on throughput.
I would agree with Brock and say check your App Pool settings first. Basically an ARR App Pool should be set to never recycle.
I would recommend watching episodes 32-38 starting at:
http://dotnetslackers.com/articles/iis/Bindings-and-Rules-for-Application-Request-Routing-ARR-Week-32.aspx
I have defined a custom LoadBalancerProbe for my webrole as follows
<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="CloudService" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" schemaVersion="2013-03.2.0">
<LoadBalancerProbes>
<LoadBalancerProbe name="MyProbe" protocol="http" intervalInSeconds="15" path="/api/ping" port="80" timeoutInSeconds="30" />
</LoadBalancerProbes>
<WebRole name="TestApp" vmsize="Small">
<Sites>
<Site name="Web">
<Bindings>
<Binding name="Endpoint1" endpointName="Endpoint1" />
</Bindings>
</Site>
</Sites>
<Endpoints>
<InputEndpoint name="Endpoint1" protocol="http" port="80" localPort="80" loadBalancerProbe="MyProbe"/>
</Endpoints>
<Imports>
<Import moduleName="Diagnostics" />
<Import moduleName="RemoteAccess" />
<Import moduleName="RemoteForwarder" />
</Imports>
</WebRole>
</ServiceDefinition>
When in Azure I have 2 instances. I have enabled trace.axd and can see the load balancer calling the ping method, so that is definitely happening.
I can also see my "503" responses (Server Unavailable) in my test app when I want my instance to appear down (I change a config setting on the instance). I can see custom HTTP Headers from the load balancer X-MS-LB-MonitorStatus Down.
When I use a Curl request to access the load-balanced url, it always returns the correct results (If I have set an instance to return 503 rather than 200, it does not appear in the response results).
When I use a browser however (in this case Chrome) I can still get results back from the instance that is supposed to be down (i.e. the instance was available, I disable it, then additional calls to the load balanced url still resolve to the "disabled" instance).
I can confirm the actual instances that resolved each request using trace.axd information
I'm struggling to believe that azure is doing load balancing properly here.
Why does the browser continue to be able to access an instance that is supposedly out of rotation?
Why does curl "always get it right"?
The Azure load balancer is a layer 3 load balancer and only load balances new incoming TCP connections. It does not know anything about HTTP traffic.
Typically a browser will establish a TCP connection with keep-alive set to true and will keep that TCP connection open for a period of time, and any subsequent requests to the website will just be HTTP traffic over the existing TCP connection. Application such as curl will typically close the TCP connection after every request.
So in your case the Azure load balancer is behaving correctly, but your browser already has a TCP connection established to the instance that is out of rotation, so future HTTP requests will still go to the same out of rotation instance.
To validate that this is what is happening you can use netmon/wireshark on the client side or the server side.
To resolve this you have a few options:
After 4 minutes of idle time the connection will be terminated
and your browser should establish a new TCP connection and get
routed to an instance that is in rotation.
Closing all browser windows and reopening the browser should establish new TCP
connections.
On the server side you can set keep-alive to false and this will terminate the TCP connection on the Azure instance.
See the 3rd Q&A at http://blogs.msdn.com/b/kwill/archive/2013/02/28/heartbeats-recovery-and-the-load-balancer.aspx for a little more information.
According to this blog post (2014) Azure uses a Layer 4 load balancer:
http://azure.microsoft.com/blog/2014/04/08/microsoft-azure-load-balancing-services/
As such it can respond and monitor HTTP 200 and 503.