Creating Azure Traffic Manager through REST API - unexpected behaviour - azure

Question

I created Traffic Manager throught its REST API using 2011-10-01 MS verion.
Resources I followed -
Create Profile -
http://msdn.microsoft.com/en-us/library/windowsazure/hh758254.aspx
Create Definition -
http://msdn.microsoft.com/en-us/library/windowsazure/hh758257.aspx
Traffic Manager got created successfully. All happies.
But after 30mins of time, traffic manager is going to INACTIVE status and all its endpoints are GONE. It shows there are no endpoints associated with it.
I am not sure what is happening around. Is it Azure problem? or is it REST API problem? or is it my way of creating Traffic manager problem.
PS - I followed this sample for making REST API calls - http://msdn.microsoft.com/en-us/library/windowsazure/gg651127.aspx
Any help would be highly appreciated.
UPDATE1
Parameters
SubscriptionID - a Valid GUID from publishsettings
Certificate - I cross checked a valid certificate present in the local cert store
endpoint1 domain name - JASH13.CLOUDAPP.NET
endpoint2 domain name - JASH23.CLOUDAPP.NET
There is no error at REST API calls level. Everything worked seamlessly.
Profile Creation -
// X.509 certificate variables.
X509Store certStore = null;
X509Certificate2Collection certCollection = null;
X509Certificate2 certificate = null;
// Request and response variables.
HttpWebRequest httpWebRequest = null;
HttpWebResponse httpWebResponse = null;
// Stream variables.
Stream responseStream = null;
StreamReader reader = null;
// URI variable.
Uri requestUri = null;
// The thumbprint for the certificate. This certificate would have been
// previously added as a management certificate within the Windows Azure management portal.
string thumbPrint = CertificateThumbprint;
// Open the certificate store for the current user.
certStore = new X509Store(StoreName.My, StoreLocation.CurrentUser);
certStore.Open(OpenFlags.ReadOnly);
// Find the certificate with the specified thumbprint.
certCollection = certStore.Certificates.Find(
X509FindType.FindByThumbprint,
thumbPrint,
false);
// Close the certificate store.
certStore.Close();
// Check to see if a matching certificate was found.
if (0 == certCollection.Count)
{
throw new Exception("No certificate found containing thumbprint " + thumbPrint);
}
// A matching certificate was found.
certificate = certCollection[0];
// Create the request.
requestUri = new Uri("https://management.core.windows.net/"
+ SubscriptionId
+ "/services/WATM/profiles");
httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(requestUri);
// Add the certificate to the request.
httpWebRequest.ClientCertificates.Add(certificate);
httpWebRequest.Method = "POST";
httpWebRequest.Headers.Add("x-ms-version", "2011-10-01");
string str = #"<Profile xmlns=""http://schemas.microsoft.com/windowsazure""><DomainName>" + ProfileDomain + "</DomainName><Name>" + ProfileName + "</Name></Profile>";
byte[] bodyStart = System.Text.Encoding.UTF8.GetBytes(str.ToString());
Stream dataStream = httpWebRequest.GetRequestStream();
dataStream.Write(bodyStart, 0, str.ToString().Length);
// Make the call using the web request.
httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();
// Parse the web response.
responseStream = httpWebResponse.GetResponseStream();
reader = new StreamReader(responseStream);
// Close the resources no longer needed.
httpWebResponse.Close();
responseStream.Close();
reader.Close();
Definition Creation-
// X.509 certificate variables.
X509Store certStore = null;
X509Certificate2Collection certCollection = null;
X509Certificate2 certificate = null;
// Request and response variables.
HttpWebRequest httpWebRequest = null;
HttpWebResponse httpWebResponse = null;
// Stream variables.
Stream responseStream = null;
StreamReader reader = null;
// URI variable.
Uri requestUri = null;
// The thumbprint for the certificate. This certificate would have been
// previously added as a management certificate within the Windows Azure management portal.
string thumbPrint = CertificateThumbprint;
// Open the certificate store for the current user.
certStore = new X509Store(StoreName.My, StoreLocation.CurrentUser);
certStore.Open(OpenFlags.ReadOnly);
// Find the certificate with the specified thumbprint.
certCollection = certStore.Certificates.Find(
X509FindType.FindByThumbprint,
thumbPrint,
false);
// Close the certificate store.
certStore.Close();
// Check to see if a matching certificate was found.
if (0 == certCollection.Count)
{
throw new Exception("No certificate found containing thumbprint " + thumbPrint);
}
// A matching certificate was found.
certificate = certCollection[0];
// Create the request.
requestUri = new Uri("https://management.core.windows.net/"
+ SubscriptionId
+ "/services/WATM/profiles/" + ProfileName + "/definitions");
httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(requestUri);
// Add the certificate to the request.
httpWebRequest.ClientCertificates.Add(certificate);
httpWebRequest.Method = "POST";
httpWebRequest.Headers.Add("x-ms-version", "2011-10-01");
string str = #"<Definition xmlns=""http://schemas.microsoft.com/windowsazure""><DnsOptions><TimeToLiveInSeconds>300</TimeToLiveInSeconds></DnsOptions><Monitors><Monitor><IntervalInSeconds>30</IntervalInSeconds><TimeoutInSeconds>10</TimeoutInSeconds><ToleratedNumberOfFailures>3</ToleratedNumberOfFailures><Protocol>HTTP</Protocol><Port>80</Port><HttpOptions><Verb>GET</Verb><RelativePath>/</RelativePath><ExpectedStatusCode>200</ExpectedStatusCode></HttpOptions></Monitor></Monitors><Policy><LoadBalancingMethod>RoundRobin</LoadBalancingMethod><Endpoints><Endpoint><DomainName>" + PrimaryService + "</DomainName><Status>Enabled</Status></Endpoint><Endpoint><DomainName>" + SecondaryService + "</DomainName><Status>Enabled</Status></Endpoint></Endpoints></Policy></Definition>";
byte[] bodyStart = System.Text.Encoding.UTF8.GetBytes(str.ToString());
Stream dataStream = httpWebRequest.GetRequestStream();
dataStream.Write(bodyStart, 0, str.ToString().Length);
// Make the call using the web request.
httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();
// Parse the web response.
responseStream = httpWebResponse.GetResponseStream();
reader = new StreamReader(responseStream);
// Close the resources no longer needed.
httpWebResponse.Close();
responseStream.Close();
reader.Close();
UPDATE2
Once the TM went into Inactive State, I checked the profile definition using REST API. In there I was not able to find any endpoints. They are missing.
<Definitions xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<Definition>
<DnsOptions>
<TimeToLiveInSeconds>300</TimeToLiveInSeconds>
</DnsOptions>
<Status>Enabled</Status>
<Version>1</Version>
<Monitors>
<Monitor>
<IntervalInSeconds>30</IntervalInSeconds>
<TimeoutInSeconds>10</TimeoutInSeconds>
<ToleratedNumberOfFailures>3</ToleratedNumberOfFailures>
<Protocol>HTTP</Protocol>
<Port>80</Port>
<HttpOptions>
<Verb>GET</Verb>
<RelativePath>/</RelativePath>
<ExpectedStatusCode>200</ExpectedStatusCode>
</HttpOptions>
</Monitor>
</Monitors>
<Policy>
<LoadBalancingMethod>Performance</LoadBalancingMethod>
<Endpoints/>
<MonitorStatus>Inactive</MonitorStatus>
</Policy>
</Definition>
</Definitions>
UPDATE3
This sporadic behavior is ONLY happening with the specific cloud services and TM profile/definitiona. When I create new set of cloud services and TM profile, then everything seems to be working fine. I tested this multiple times. So the only problem is with following parameters.
endpoint1 domain name - JASH13.CLOUDAPP.NET
endpoint2 domain name -JASH23.CLOUDAPP.NET
TM Domain - ramitm.trafficmanager.net
TM profilename - ramitm

Answer 1

This seems like some DNS Problems for very fast REST API Operations. I was not able to get the crux of the problem, but this is how I solved it.
Previously I was getting this problem for this patter - Create -> Delete -> Re-Create
Now I made it this way - Create -> Delete -> Delay -> Re-Create
I think by introducing delay component, I am giving enough time for Azure to settle down all the DNS and infrastructure and there by update them. So after introducing delay, I was not experiencing the problem. Delay can be 5 - 10 mins.