I want to use deployment slots for my Blazor server side application, but it stops working for the current users during the swap and they have to refresh the page.
I'm using an Azure SignalR Service for performance reasons, so it kinda makes sense, I imagine it like this:
Connections are held in memory and when I swap, obviously that is gone. At least without a SignalR Service. But shouldn't my SignalR Service keep SignalR connections (see red)? Did I set it up incorrectly?
I found others having similar problems (without using Blazor), but I'm not sure if these are viable with Blazor, especially because I just want to mitigate that 1-2 minute downtime for an update...
Automated reconnect
SignalR client disconnected on Azure slots swap
Storing connections in an external storage. But manually handling connections is absurd effort?
https://learn.microsoft.com/en-us/aspnet/signalr/overview/guide-to-the-api/mapping-users-to-connections
Update:
See: https://www.youtube.com/watch?v=Vvjdqq8MB44&t=12m10s
It seems there is "Web traffic" going directly to the Blazor app. My guess: After a swap the "Web traffic" still goes to the previous instance, while SignalR traffic goes to the newly swapped in instance. That sounds like a problem.
But once again, I have no clue what "Web traffic" actually is or if that is the problem and if Azure offers a way to solve the issue, so a definitive answer would still be appreciated.
I don't think you have it setup incorrectly. I looked into doing a similar thing and had the same results.
The azure signalr service is basically acting as a proxy, when you switch slot the azure signalr server is losing its connection to the blazor server hub which holds the current state.
I don't think there is anyway around it, when you want to update your blazor server site every connected client will lose its connection because it's not possible to move the client 'states' over to the new slot/site.
What would be nice is the ability for azure to switch the new slot in once all old connections have disconnected, but I don't think we'll ever get that as it's a very specific blazor server requirement.
Related
I have hosted an MVC application in Azure App Service. The application is not creating any problem when i am testing it in local host. I can easily connect to the Dynamics-365 through the SDK and can do easily CRUD operation.
But when i am trying to login which is hosted in Azure it is giving that error though am not getting this error if i restart the app service. If i restart the app service in every morning then i am not getting this error. But i don't want to restart the app service daily.
Here the same question also has been asked but there is not any solution which has been marked as Solved. Time zone difference between client and server has been discussed here but the part which confused me when i am restarting the app service it is working.
How to solve this issue? I am thinking about app service Auto Healing. Should this work?
Any solution would be appreciated.
Thanks in advance.
From your description of Question it looks like connection pool issue. You probably are not closing connection after your transaction is completed. Try force closing connection (threads) that might solve your issue.
OR
The clock on the Server hosting client application was out of sync with the server having the service.
This is almost always because of a server time skew. The remote server
and the client's system time must be within (typically) 10 minutes of
each other. If they are not, security validation will fail.
I'd check azure service bus and find out what their server time is, and
compare that to your server time.
WCF gives an unsecured or incorrectly secured fault error
https://social.msdn.microsoft.com/Forums/vstudio/en-US/ae4172dd-e215-4a72-b927-d164ce4cc318/an-unsecured-or-incorrectly-secured-fault-was-received-from-the-other-party?forum=wcf
I have a NodeJS server running on an Azure free website. The server has a websocket module installed. Each connected user will cache some data with an object so that anyone else who connects can retrieve cached data from this object. The problem I am experiencing is that the server doesn't seem to keep this object around for very long. I can access the data with in for some time, but if I try later in the day, it's just gone.
Is Azure shutting down the server because it is experiencing no activity, causing the object to be deallocated? Does NodeJS deallocate objects if they aren't used after some time?
Azure Websites, as Ben pointed out in his answer, will evict idle websites. This is especially true with free/shared tiers, since your website is sharing resources with several other tenants on the same VM instances. But even with basic and standard tiers, there may be a need to evict your website (especially since you can have many of your own websites running within a single hosting plan).
With basic/standard tier websites, you have the ability to enable Always On. You'll see this option under the Configure tab:
Once you enable this, your website should remain loaded.
Yep. If there aren't any requests to the app pool, Azure Websites stops your application. That means anything in memory is lost. You can set up a cron job or scheduled task to ping your web app to avoid the app pool timing out due to inactivity.
EDIT: Or, as David Makogon pointed out,
With basic/standard tier websites, you have the ability to enable Always On. You'll see this option under the Configure tab:
We are running some long-running test apps with Azure Service Bus relay over http, hosted in a windows service and most of the time, these run fine for 2-3 days. However every so often an internal network glich may occur (e.g. firewall reboots) that kills the internet connection.
At this point, the relay is dropped in Azure and our web app can no longer communicate with the on-premise service.
I would have thought that the Azure relay client was fault-tolerant - in that if it realises that it's lost connection with Azure then it will re-establish the connection andf if it can't keep trying until it can.. but it appears that this is not the case. This seems pretty fundamental...?
Only once have I ever seen a "System.ServiceModel.CommunicationException" where the service can't communicate on the internet, and that was when the client was starting up and trying to establish the connection in the first place.
Is there any advice or feedback on handling transient disconnections through the relay service (as it's a cloud --> on-prem direction then the client can't AFAIK ping the server).
If you are still experiencing issues, you may want to contact Azure support to understand why it is disconnecting. The Relay client should reconnect if something happens to the existing connection.
You may want to add ConnectionStatusBehavior to your ChannelFactory to have it output when the status for the connection changes. It will contain the error that caused it to change status.
var connectionStatusBehavior = new ConnectionStatusBehavior();
connectionStatusBehavior.Online += ConnectionStatusOnlineMethod;
connectionStatusBehavior.Offline += ConnectionStatusOfflineMethod;
channelFactory.Endpoint.Behaviors.Add(connectionStatusBehavior);
This issue is solved by Microsoft in version 2.6.5 of Microsoft Azure Service Bus dll. After 1 month of testing it seems to work.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
We are running a self hosted AppService with ServiceStack 3.x
We would like to have a automatic failover mechanism on the clients if the current service running as master fails.
Clients at the moment are strong typed C# using the default SS JSONClient, but we will add web based clients (AngularJS) in the future.
Does anybody have an idea, how that could be done?
Server side redundancy & failover:
That's a very broad question. A ServiceStack self hosted application is no different to any other web-facing resource. So you can treat it like a website.
Website Uptime Monitoring Services:
You can monitor it with regular website monitoring tools. These tools could be as simple as an uptime monitoring site that simply pings your web service at regular intervals to determine if it up, and if not take an action, such as triggering a restart of your server, or simply send you an email to say it's not working.
Cloud Service Providers:
If you are using a cloud provider such as Amazon EC2, they provide CloudWatch services that can be configured to monitor the health of your host machine and the Service. In the event of failure, it could restart your instance, or spin up another instance. Other providers provide similar tools.
DNS Failover:
You can also consider DNS failover. Many DNS providers can monitor service uptime, and in the event of a failover their service will change the DNS route to point to another standby service. So the failover will be transparent to the client.
Load Balancers:
Another option is to put your service behind a load balancer and have multiple instances running your service. The likelihood of all the nodes behind the load balancer failing is usually low, unless there is some catastrophically wrong with your service design.
Watchdog Applications:
As you are using a self hosted application, you may consider making another application on your system that simply checks that your service application host is running, and if not restarts it. This will handle cases where an exception has caused you app to terminate unexpectedly - of course this is not a long term solution, you will need to fix the exception.
High Availability Proxies (HAProxy, NGINX etc):
If you are run your ServiceStack application using Mono on a Linux platform there are many High Availability solutions, including HAProxy or NGINX. If you run on a Windows Server, they provide failover mechanisms.
Considerations:
The right solution will depend on your environment, your project budget, how quickly you need the failover to resolve. The ultimate considerations should be where will the service failover to?
Will you have another server running your service, simply on standby - just in case?
Will you use the cloud to start up another instance on demand?
Will you try and recover the existing application server?
Resources:
There are lots of articles out there about failover of websites, as your web service use HTTP like a website, they will also apply here. You should research into High Availability.
Amazon AWS has a lot of solutions to help with failover. Their Route 53 service is very good in this area, as are their loadbalancers.
Client side failover:
Client side failover is rarely practical. In your clients you can ultimately only ever test for connectivity.
Connectivity Checking:
When connectivity to your service fails you'll get an exception. Upon getting the exception, the only solution would be to change the target service URL, and retry the request. But there are a number of problems with this:
It can be as expensive as server side failover, as you have to keep the failover service(s) online all the time for the just-in-case moments. Some server side solutions would allow you to start up a failover service on demand, thus reducing cost significantly.
All clients must be aware of the URL(s) to failover too. If you managed the failover at DNS i.e. server side then clients wouldn't have to worry about this complexity.
Your client can only see connectivity failures, there may not be an issue with the server, it may be their connectivity. Imagine the client wifi goes down for a few seconds while servicing your request to your primary service server. During that time the client gets the connectivity exception and you try to send the request to the failover secondary service server, at which point their wifi comes online. Now you have clients using both the primary and secondary service. So their network connectivity issues become your data consistency problems.
If you are planning web based clients, then you will have to setup CORS support on the server, and all clients will require compatible browsers, so they can change the target service URL. CORS requests have the disadvantages of having more overhead that regular requests, because the client has to send OPTIONS requests too.
Connectivity error detection in clients is rarely fast. Sometimes it can take in excess of 30 seconds before a client times out a request as having failed.
If your service API is public, then you rely on the end-user implementing the failover mechanism. You can't guarantee they will do so, or that they will do so correctly, or that they won't take advantage of knowing your other service URLs and send requests there instead. Besides it look very unprofessional.
You can't guarantee that the failover will work when needed. It's difficult to guarantee that for any system, even big companies have issues with failover. Server side failover solutions sometimes fail to work properly but it's even more true for client side solutions because you can test the failover solution ahead of time, under all the different client side environmental factors. Just because your implementation of failover in the client worked in your deployment, will it work in all deployments? The point of the failover solution after all is to minimise risk. The risk of server side failover not working is far less than client, because it's a smaller controllable environment, which you can test.
Summary:
So while my considerations may not be favourable of client side failover, if you were going to do it, it's a case of catching connectivity exceptions, and deciding how to handle them. You may want to wait a few seconds and retry your request to the primary server before immediately swapping to the secondary just in case it was an intermittent error.
So:
Catch the connectivity exception
Retry the request (maybe after a small delay)
Still failing, change the target host and retry
If that fails, it's probably a client connectivity issue.
I have a SignalR service running on an Azure Web Role. I'm interested in deploying a new version of the service, first to Staging and then to the Production deployment slot.
My question is if I swap the two deployment slots while there are existing connections to the service on Production:
What will happen to those existing connections? Will they still be connected to the now-Staging slot, or will they reconnect to the the now-Production slot? Or something else? Is this client-dependent?
Thanks!
It's going to depend on how you're storing SignalR connection information.
If you're using a permanent external storage solution (see this post), then swapping from one web server to another, which is what the VIP swap will end up doing, shouldn't interrupt the connections.
If you're using a different approach for storing the connection info like in-memory, then it's going to be lost b/c you're going to be transferring client traffic from one web server to another with the deployment swap.