Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
We are running a self hosted AppService with ServiceStack 3.x
We would like to have a automatic failover mechanism on the clients if the current service running as master fails.
Clients at the moment are strong typed C# using the default SS JSONClient, but we will add web based clients (AngularJS) in the future.
Does anybody have an idea, how that could be done?
Server side redundancy & failover:
That's a very broad question. A ServiceStack self hosted application is no different to any other web-facing resource. So you can treat it like a website.
Website Uptime Monitoring Services:
You can monitor it with regular website monitoring tools. These tools could be as simple as an uptime monitoring site that simply pings your web service at regular intervals to determine if it up, and if not take an action, such as triggering a restart of your server, or simply send you an email to say it's not working.
Cloud Service Providers:
If you are using a cloud provider such as Amazon EC2, they provide CloudWatch services that can be configured to monitor the health of your host machine and the Service. In the event of failure, it could restart your instance, or spin up another instance. Other providers provide similar tools.
DNS Failover:
You can also consider DNS failover. Many DNS providers can monitor service uptime, and in the event of a failover their service will change the DNS route to point to another standby service. So the failover will be transparent to the client.
Load Balancers:
Another option is to put your service behind a load balancer and have multiple instances running your service. The likelihood of all the nodes behind the load balancer failing is usually low, unless there is some catastrophically wrong with your service design.
Watchdog Applications:
As you are using a self hosted application, you may consider making another application on your system that simply checks that your service application host is running, and if not restarts it. This will handle cases where an exception has caused you app to terminate unexpectedly - of course this is not a long term solution, you will need to fix the exception.
High Availability Proxies (HAProxy, NGINX etc):
If you are run your ServiceStack application using Mono on a Linux platform there are many High Availability solutions, including HAProxy or NGINX. If you run on a Windows Server, they provide failover mechanisms.
Considerations:
The right solution will depend on your environment, your project budget, how quickly you need the failover to resolve. The ultimate considerations should be where will the service failover to?
Will you have another server running your service, simply on standby - just in case?
Will you use the cloud to start up another instance on demand?
Will you try and recover the existing application server?
Resources:
There are lots of articles out there about failover of websites, as your web service use HTTP like a website, they will also apply here. You should research into High Availability.
Amazon AWS has a lot of solutions to help with failover. Their Route 53 service is very good in this area, as are their loadbalancers.
Client side failover:
Client side failover is rarely practical. In your clients you can ultimately only ever test for connectivity.
Connectivity Checking:
When connectivity to your service fails you'll get an exception. Upon getting the exception, the only solution would be to change the target service URL, and retry the request. But there are a number of problems with this:
It can be as expensive as server side failover, as you have to keep the failover service(s) online all the time for the just-in-case moments. Some server side solutions would allow you to start up a failover service on demand, thus reducing cost significantly.
All clients must be aware of the URL(s) to failover too. If you managed the failover at DNS i.e. server side then clients wouldn't have to worry about this complexity.
Your client can only see connectivity failures, there may not be an issue with the server, it may be their connectivity. Imagine the client wifi goes down for a few seconds while servicing your request to your primary service server. During that time the client gets the connectivity exception and you try to send the request to the failover secondary service server, at which point their wifi comes online. Now you have clients using both the primary and secondary service. So their network connectivity issues become your data consistency problems.
If you are planning web based clients, then you will have to setup CORS support on the server, and all clients will require compatible browsers, so they can change the target service URL. CORS requests have the disadvantages of having more overhead that regular requests, because the client has to send OPTIONS requests too.
Connectivity error detection in clients is rarely fast. Sometimes it can take in excess of 30 seconds before a client times out a request as having failed.
If your service API is public, then you rely on the end-user implementing the failover mechanism. You can't guarantee they will do so, or that they will do so correctly, or that they won't take advantage of knowing your other service URLs and send requests there instead. Besides it look very unprofessional.
You can't guarantee that the failover will work when needed. It's difficult to guarantee that for any system, even big companies have issues with failover. Server side failover solutions sometimes fail to work properly but it's even more true for client side solutions because you can test the failover solution ahead of time, under all the different client side environmental factors. Just because your implementation of failover in the client worked in your deployment, will it work in all deployments? The point of the failover solution after all is to minimise risk. The risk of server side failover not working is far less than client, because it's a smaller controllable environment, which you can test.
Summary:
So while my considerations may not be favourable of client side failover, if you were going to do it, it's a case of catching connectivity exceptions, and deciding how to handle them. You may want to wait a few seconds and retry your request to the primary server before immediately swapping to the secondary just in case it was an intermittent error.
So:
Catch the connectivity exception
Retry the request (maybe after a small delay)
Still failing, change the target host and retry
If that fails, it's probably a client connectivity issue.
Related
I'm using the Hybrid Connection Manager and also the On Premise Data Gateway for several projects hosted in the Azure cloud.
There are more and more use cases for those two components and I need to setup a clean monitoring to detect connection troubles (for example when there is a network issue or a reboot of the servers hosting the gateways).
For the HCM, there are Relays metrics I can rely on, but I saw that some of those counters are not reliable. I had issues with my connexion in the past few days, and when I check the ListenerConnections-ClientError or ListenerConnections-ServerError counters, they always equal to 0... this sounds very strange?
Regarding the OnPremise Data Gateway, I think that because it also relies on SBus Relay, I should probably use the same metrics?
I just created my first Azure SQL Server and Database and now am trying to configure the firewall so that my web application can connect and make changes to the single database on the server.
I see that you can allow all clients to connect by allowing a rule such:
However is this bad practice? I see my Client IP address is listed in the Azure portal can I get clarification that I should allow just that single IP address access for now, and then later when I publish my web application to Azure I will be given an IP address of where that web app lives and only restrict access to that (assuming that people can only make database changes through my front end application).
Thanks
Yes, this is bad practice. There are other layers they'd have to get through (e.g., the server login), but this opens the front door and allows anyone to pick away at it at their leisure.
If you have a web server hosting your web application on some server (which you must be), whitelist that server's IP address (and perhaps your own, for development/admin purposes), but allowing all IPs is not considered good practice, no.
One particular case where you may really want to allow this is if you are distributing a desktop application to unknown clients that must connect to the backend. It becomes extremely enticing at that point, but even so, the recommended practice (or at least, my recommended practice) would be to utilize a web service that will accept an application registration upon startup of the app, register the client IP temporarily through that, and then have a background server (think WebJobs) that will flush the firewall rules say, every night or so (or for a more elaborate setup, accept the registration with a timeout and use a continuous WebJob to poll for the timeout and refresh/revoke as necessary)
I have a mobile application that communicates with a REST based web-service. The web-service lives behind the firewall and talks to other systems. Currently this web-service requires a firewall port to be opened and a SSL cert generated for each installation. Mobile apps sends login credentials so web-services can login to custom back-end systems.
Recently a customer approached us asking how could we deploy this to 50 offices. As we don't want to say modify every firewall in every office, we're looking for options.. This is a list of possible solutions and my thoughts on each one:
Open firewall port and expose https webservice - This is our current
solution but we dont want to have to contact 50 network admins and explain why we need to do this.
VPN - Too heavy weight, complex and expensive, we only need access
to one server. Does not solve problem as firewall needs to be
modified.
Microsoft Azure Hybrid Connection Manager - This provides a managed
service where the Azure cloud will expose an end point. Azure will
also expect connections from a easy to install application that
lives behind the firewall. When a REST call is made to the cloud
end-point, the request is forward down socket that was initiated by
the software behind the firewall. This does what we want but as its
a Microsoft Solution there might impose other requirements that our
customers might not want. Currently the simple Hybrid Connection Manager is free. But for how long?
Jscape MFT Gateway - Similar to Azure but you can host their server anywhere. Not that expensive but is not opensource.
Netty - A async java library/toolkit where this type of application could easily be build. Client and server apps would need to be build and deployed. Dont know what we dont know about Netty.
MDM, AirWatch, BlackBerry BES - A MDM based solution would work expect that MDM's are centrally managed and are not often in every office where the backend services are located. Airwatch has an AppTunnle but im not sure about the specifics.
At this point the Microsoft and Jscape systems are possible solutions.
But most likely these solutions will require us to modify the mobile software to work around issues such as:
How does the user know which server to login to? A locator service
needs to be built such that, an email address is used to lookup their
office, or they need to select their office location from a list.
While the connection is SSL many company might want some additional protection since network login information will be send down the pipe.
How is load balancing and fail-over managed?
So, at this point i'm looking for more options. The best option would be a commercial product that offers some level of customization. Second, would like a well used open-source product that could be installed in Aws and customized.
Thanks
The best approach we found was to use the PUTTY API and setup a reverse proxy.
If I deploy a web app (formerly known as an Azure WebSite) to an App Hosting Plan in Azure with a couple of instances (scale = 2) will the load balancer in front of the instances care if any of the instances is unhealthy?
I'm troubleshooting an issue that sometimes causes my site to return an http 503 ~50% of the time. My thinking here is that one of two of my instances has failed but the load balancer hasn't noticed.
If the load balancer does care, what does it look for? I can't find anyway to specify a ping url, for instance.
Note: this question has nothing to do with Traffic Manager.
Yes, Azure Web Apps monitors the health of the workers by making internal requests to it and verifying that they're healthy.
However, we don't check status codes that the web app returns to user requests (like 500, etc) since that could easily be an app specific issue rather than a problem with the machine.
So the answer you're looking for is: We continuously test whether or not the instances (VMs) are healthy and take them down if they're not. However, those tests do not rely on error codes the customer's site returns
We integrate with a third-party service where we can run queries which is right now secured using HTTPS encryption and username/password. We send our queries from a service running on the Windows Azure cloud.
The third-party provider wants to migrate towards better security and they have asked us to either
Setup a VPN - which is problematic because for we'd need to use Azure Connect and they'd have to install the client endpoint service on their part.
Provide some IP address where the queries will come from so they can filter out anyone else at the firewall level - which is problematic because AFAIK you cannot fix the IP addresses of the Windows Azure Compute nodes.
Suggest another secure alternative - the only thing I could think of is to set up the VPN with them on a non-Azure server and then tunnel the requests through using Azure Connect - which is obviously extra work for us and also defeats the point of hosting the service on a cloud if it depends on a non-cloud service.
Any ideas?
Can they install the Azure Connect endpoint on another server on their DMZ network? i.e. not the actual server which hosts their service?
Can we somehow provide them with static IPs for incoming queries?
Any other solution that is scalable?
Thanks
If I understand the scenario correctly, your Azure service is a client to a 3rd party service. This scenario may be solved through the use of the Windows Azure AppFabric Service Bus. You would need to install a proxy app in the 3rd party's datacenter that would be responsible for establishing the connection to the service bus. The connection comes from inside the 3rd party's datacenter, so no new incoming holes in the firewall. The connection can handle WCF connections with all its security strengths, and users can be authenticated with ACS.
Here is a starting point: http://msdn.microsoft.com/en-us/library/ee732537.aspx
There is a hands on lab in the Windows Azure Platform Training Kit that explains most of the details that you'll need.
IMHO, HTTPS is already very good; and I don't exactly see how a VPN would make the system any more secure. In particular, VPN is no silver bullet, if your VM is compromised then the VPN connection is compromised too (same for HTTPS). On the other hand, the IP restriction would indeed reduce the attack surface.
Then, using a server outside the cloud is a poor idea indeed. Not only it defeats most of the benefits of the cloud (been there, done that and suffered a lot), but also it also makes the whole thing less secure with more complexity and more attack surface.
Windows Azure does not provide anything that look like a static IP at this point. In our experience, IP addresses for a given service change once in a while even if the service is only upgraded (and never deleted). Static IP addresses have been an important feature request for a long time, Microsoft will probably provide it at some point, but it might still take many months.