I'm using the Azure Dedicated cache to store my session state in and I've recently been getting connection failures which then take my application down. The only way to fix it is to restart the app.
Does anyone know of a way to catch this type of exception and then refresh the cache connection on the fly?
I am using azure caching library 2.3 and I following this article to setup the sessionstate in the cache.
http://azure.microsoft.com/en-us/documentation/articles/cache-dotnet-how-to-use-service/#store-session
Here is the exception I am getting.
"ClassName": "Microsoft.ApplicationServer.Caching.DataCacheException",
"Message": "There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.)",
"Data": {
"The client was trying to communicate with the server: net.tcp://xxxxx.cache.windows.net:23233.": null
},
Thanks!
I was directed to this post which gives some details on how to refresh the Azure DataCacheFactory.
http://blogs.msdn.com/b/cie/archive/2014/04/29/cache-retry-fails-what-next.aspx
I'm only using the built in SessionProvider so I think this was a bit much and I couldn't understand everything that was going on. So instead I'm catching the exception and then restarting the role so that a new connection can be established on app startup.
Here is the root cause of problem from that post..
Reasons in general can be in case of High Availability the underlying
cache service is load balancing the partitions and the secondary node
is transitioning to primary and the client still is sending request to
old primary node OR for some reason the cache service got moved to a
different VM as part of service healing process but cache client still
is having the old IP address of cache service VM.
Though its good to have a retry policy in place but in extreme cases
where retry is not helping then you could use below approach in your
application to mitigate the errors by refreshing the cache client when
an exception is thrown.
Related
Created an azure MVC website, from service (controller) code we are connecting to an on-premise sql server using Azure Hybrid Connection. Intermittently we are facing below issue.
"A transport-level error has occurred when receiving results from the
server. (provider: TCP Provider, error: 0 - The specified network name
is no longer available.)"
Please provide suggestions to resolve this issue.
You can try following solutions :
Try increasing connection time-out.
check if remote connection is enabled.
Try adding firewall exception.
First of all the error means either the networks has some extra latency, the database is down or you may have too many concurrent connections open the database.
(Make sure you are closing all open datareaders.)
also it may be due to this
These are transient faults and are to be expected in the cloud. Implementing defensive programming is usually a must in the cloud. Try using some retry logic. Microsoft's transient fault exception library is an excellent start. Though meant primarily for SQL Azure and Azure Service bus, you can use the library for SQL IaaS.
In my opinion, 98% sure, because I recently had the same experience, it is a network issue from the server provider.
For instance: if you are rent the server from Ionos, by default all remote connections are blocked, even though you disable the firewall in the server. You still won't be able to connect remotely. You can, however, do your work on the server without any problem.
To connect remotely, you have to contact the server provider. They will explain how to enable firewall ports from your control panel.
I contacted my server provider as I almost get frustrated. Here was their response.
enter image description here
After this, every permitted client can connect remotely to the server.
I wish you success.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
We are running a self hosted AppService with ServiceStack 3.x
We would like to have a automatic failover mechanism on the clients if the current service running as master fails.
Clients at the moment are strong typed C# using the default SS JSONClient, but we will add web based clients (AngularJS) in the future.
Does anybody have an idea, how that could be done?
Server side redundancy & failover:
That's a very broad question. A ServiceStack self hosted application is no different to any other web-facing resource. So you can treat it like a website.
Website Uptime Monitoring Services:
You can monitor it with regular website monitoring tools. These tools could be as simple as an uptime monitoring site that simply pings your web service at regular intervals to determine if it up, and if not take an action, such as triggering a restart of your server, or simply send you an email to say it's not working.
Cloud Service Providers:
If you are using a cloud provider such as Amazon EC2, they provide CloudWatch services that can be configured to monitor the health of your host machine and the Service. In the event of failure, it could restart your instance, or spin up another instance. Other providers provide similar tools.
DNS Failover:
You can also consider DNS failover. Many DNS providers can monitor service uptime, and in the event of a failover their service will change the DNS route to point to another standby service. So the failover will be transparent to the client.
Load Balancers:
Another option is to put your service behind a load balancer and have multiple instances running your service. The likelihood of all the nodes behind the load balancer failing is usually low, unless there is some catastrophically wrong with your service design.
Watchdog Applications:
As you are using a self hosted application, you may consider making another application on your system that simply checks that your service application host is running, and if not restarts it. This will handle cases where an exception has caused you app to terminate unexpectedly - of course this is not a long term solution, you will need to fix the exception.
High Availability Proxies (HAProxy, NGINX etc):
If you are run your ServiceStack application using Mono on a Linux platform there are many High Availability solutions, including HAProxy or NGINX. If you run on a Windows Server, they provide failover mechanisms.
Considerations:
The right solution will depend on your environment, your project budget, how quickly you need the failover to resolve. The ultimate considerations should be where will the service failover to?
Will you have another server running your service, simply on standby - just in case?
Will you use the cloud to start up another instance on demand?
Will you try and recover the existing application server?
Resources:
There are lots of articles out there about failover of websites, as your web service use HTTP like a website, they will also apply here. You should research into High Availability.
Amazon AWS has a lot of solutions to help with failover. Their Route 53 service is very good in this area, as are their loadbalancers.
Client side failover:
Client side failover is rarely practical. In your clients you can ultimately only ever test for connectivity.
Connectivity Checking:
When connectivity to your service fails you'll get an exception. Upon getting the exception, the only solution would be to change the target service URL, and retry the request. But there are a number of problems with this:
It can be as expensive as server side failover, as you have to keep the failover service(s) online all the time for the just-in-case moments. Some server side solutions would allow you to start up a failover service on demand, thus reducing cost significantly.
All clients must be aware of the URL(s) to failover too. If you managed the failover at DNS i.e. server side then clients wouldn't have to worry about this complexity.
Your client can only see connectivity failures, there may not be an issue with the server, it may be their connectivity. Imagine the client wifi goes down for a few seconds while servicing your request to your primary service server. During that time the client gets the connectivity exception and you try to send the request to the failover secondary service server, at which point their wifi comes online. Now you have clients using both the primary and secondary service. So their network connectivity issues become your data consistency problems.
If you are planning web based clients, then you will have to setup CORS support on the server, and all clients will require compatible browsers, so they can change the target service URL. CORS requests have the disadvantages of having more overhead that regular requests, because the client has to send OPTIONS requests too.
Connectivity error detection in clients is rarely fast. Sometimes it can take in excess of 30 seconds before a client times out a request as having failed.
If your service API is public, then you rely on the end-user implementing the failover mechanism. You can't guarantee they will do so, or that they will do so correctly, or that they won't take advantage of knowing your other service URLs and send requests there instead. Besides it look very unprofessional.
You can't guarantee that the failover will work when needed. It's difficult to guarantee that for any system, even big companies have issues with failover. Server side failover solutions sometimes fail to work properly but it's even more true for client side solutions because you can test the failover solution ahead of time, under all the different client side environmental factors. Just because your implementation of failover in the client worked in your deployment, will it work in all deployments? The point of the failover solution after all is to minimise risk. The risk of server side failover not working is far less than client, because it's a smaller controllable environment, which you can test.
Summary:
So while my considerations may not be favourable of client side failover, if you were going to do it, it's a case of catching connectivity exceptions, and deciding how to handle them. You may want to wait a few seconds and retry your request to the primary server before immediately swapping to the secondary just in case it was an intermittent error.
So:
Catch the connectivity exception
Retry the request (maybe after a small delay)
Still failing, change the target host and retry
If that fails, it's probably a client connectivity issue.
I am having a WCF web service and i am uploading a document to SharePoint site through this service.In my staging server it works fine with a single server but in the production we are having four servers in Load balancing and it is giving me the error...
An error occurred while receiving the HTTP response to http://jai-dms-app.rajdiscoms.com:1111/_vti_bin/Discom/Service.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.
Please help me ..
The reason why you are having an issue is most likely due to the loop back check. You will need to update the registry to fix that
http://support.microsoft.com/kb/926642/en-us
You may also want to ensure your load balance has sticky sessions
I have an MVC 4 website using a WCF service. When I deploy to Windows Azure using the VS 2012 publish wizard, I get this error:
10:13:19 AM - The HTTP request to 'https://management.core.windows.net/42d4257b-5f38-400d-aac5-2e7acee9597d/services/hostedservices/myapp?embed-detail=true' has exceeded the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout.
After cleaning the project and publishing a few times, the error goes away. What am I doing wrong?
Whenever you start publish process from VS machine, a SSL tunnel is established first and once the tunnel is created, the package is transferred from your machine to Windows Azure Portal first. After the upload is completed, you will see the result notifications are posted back to Publish result windows and that is how it happens.
In your case, the time to build the SSL tunnel doe secure package transfer is longer then normal, this could be because of network latency between your machine and the Windows Azure Management Portal. For security reason the time to create the tunnel smaller windows and if the connection is not created, the retry cycle starts the process again and even if that fails you are greeted with the failure message. This could be caused by excessive traffic on either side or both sides. So this is mainly a networking related issue rather then specific to Windows Azure as after some time successive tries, you could upload your package.
In such failure/situation, you can run network capture utilities i.e netmon, wireshark, and see the time taken during failure and success to see the different in various transfer. This will help you to understand the underlying delaying issues.
Try to update your roles diagnostics
like below
then update your storage credentials because it may be expired.
I am getting below exception when I access Azure cache from my dev azure application.
ErrorCode:SubStatus:There is a temporary failure.
Please retry later. (One or more specified cache servers are
unavailable, which could be caused by busy network or servers. For
on-premises cache clusters, also verify the following conditions.
Ensure that security permission has been granted for this client
account, and check that the AppFabric Caching Service is allowed
through the firewall on all cache hosts. Also the MaxBufferSize on the
server must be greater than or equal to the serialized object size
sent from the client.)
Is there any restriction that the Azure cache can only be accessed from hosted Azure application?
You should have no issue accessing AppFabric Cache from the emulator. Just note that the cache is only in Windows Azure itself; there's no emulated cache running locally.
Do you have any further detail on the error message? Also, are you accessing it directly in code? Using it as a session state provider?
Yes we can access Azure AppFabric cache from emulators. Just want to configure correctly and there should be no firewall blocking