unable to fetch azure resource's metrics data - azure

I have an azure function .net core 3.1 where I am trying to fetch metrics data
if I use a simpler code
string queryString = "name.value eq 'CpuPercentage'";
ODataQuery<MetadataValue> odataQuery = new ODataQuery<MetadataValue>(queryString);
string resId = app.Id;
var vals = await monitoringClient.Metrics.ListAsync(resourceUri: "/subscriptions/xxxxx/resourceGroups/xxxx/providers/Microsoft.Web/sites/xxxxxx", odataQuery: odataQuery, cancellationToken: CancellationToken.None);
i get the error "resource not found" however am able to see the resource on portal and metrics data is also available on the portal . I registered an app in AD and gave it a readers access on subscription.
when i use
var metricNames = "name.value eq 'CpuPercentage'";
// Time grain is optional when metricNames is present
string timeGrain = "timeGrain eq duration'PT5M'";
//string endDate = string.Format(" and endTime eq {0}", DateTime.Now.ToString("o"));
string startDate = string.Format(" and startTime eq {0}", DateTime.Now.AddHours(-3).ToString("o")).Replace("+", "%2b");
string endDate = string.Format(" and endTime eq {0}", DateTime.Now.ToString("o")).Replace("+", "%2b");
var odataFilterMetrics = new ODataQuery<MetadataValue>(
string.Format(
"{0}{1}{2}{3}",
metricNames,
timeGrain,
startDate,
endDate));
var metrics = await monitoringClient.Metrics.ListAsync(resourceUri: app.Id, odataQuery: odataFilterMetrics, cancellationToken: CancellationToken.None);
i get
only supports conditions of the form ' eq ', ' ne ' and ' sw ': name eq 'CpuPercentage'timeGrain eq duration'PT5M'
i have also tried various dateformats
// Defaulting to 3 hours before the time of execution for these datetimes
//string startDate = string.Format(" and startTime eq {0}", DateTime.Now.AddHours(-3).ToString("o"));
//string endDate = string.Format(" and endTime eq {0}", DateTime.Now.ToString("o"));
string startDate = string.Format(" and startTime eq {0}", DateTime.Now.AddHours(-3).ToString("o")).Replace("+", "%2b");
string endDate = string.Format(" and endTime eq {0}", DateTime.Now.ToString("o")).Replace("+", "%2b");
i am creating the monitoring client like this
private async Task<MonitorManagementClient> GetMonitorClientAsync()
{
var serviceCreds = await ApplicationTokenProvider.LoginSilentAsync("xxx.onmicrosoft.com", "xxx-appId-registered-in-AD", "xx-secret-of-that-app");
var monitorClient = new MonitorManagementClient(serviceCreds);
monitorClient.SubscriptionId = "xxx";
return monitorClient;
}

Related

Azure table storage API inconsistent replies

I have an http-triggered azure function that receives a zip code. It then queries an Azure Table using the table API and retrieves the city, state, etc. The table contains zip/postal codes for US and CAN so there's about a million rows. When I send a request it returns the correct value the first time but if I keep sending it over and over it randomly switches between returning the record and returning an empty set. So it's not failing, and I'm not getting any kind of error like a timeout.
here is an example of a successful reply:
{
odata.metadata: "https://storageaccount###.table.core.windows.net/$metadata#Table###",
value: [
{
odata.etag: "W/"datetime'2019-10-18T16%3A02%3A26.9420514Z'"",
PartitionKey: "Portsmouth",
RowKey: "00210",
Timestamp: "2019-10-18T16:02:26.9420514Z",
AreaCode: "603",
City: "Portsmouth",
Country: "US",
Pref: "P",
State: "NH",
Zip: "00210"
}
]
}
and here is an empty one after pressing F5 after getting above reply:
{
odata.metadata: "https://storageaccount###.table.core.windows.net/$metadata#Table###",
value: [ ]
}
And then if I keep pressing F5 sometimes I get the record and sometimes I don't.
here are the table api url parameters (SAS-minus the signature)
?$filter=RowKey eq '00210' and Country eq 'US'
&sv=2019-02-02
&ss=t
&srt=sco
&sp=r
&se=2099-10-18T05:27:30Z
&st=2019-10-17T21:27:30Z
&spr=https
Does anyone know why it's behaving this way or what I could look into to figure it out?
According to this page there is a 5 second timeout for querying azure tables
(https://learn.microsoft.com/en-us/rest/api/storageservices/query-timeout-and-pagination). but when I look at the headers in postman I don't see any tokens.
postman results: https://i.stack.imgur.com/hReDM.png
full CORRECTED code
public static async Task<string> Run(HttpRequest req, ILogger log)
{
log.LogInformation("Start time = "+ DateTime.Now);
string apiResponse = "";
string zip = req.Query["zip"];
if(string.IsNullOrEmpty(zip)){return "No zip code found - please provide a url parameter in the format 'zip=[code]'";}
string apiBaseUrl = "https://***storage.table.core.windows.net/zip**?";
string queryFilter = "$first&$filter=RowKey eq '" + zip + "'";
//generate auth url in storage account in Azure
string authorization = "&sv=2019-02-02&ss=t&srt=sco&sp=r&se=2099-10-18T00:38:11Z&st=2019-10-17T16:38:11Z&spr=https&sig=7S%2BkaiTwGsZIkL***";
Regex rx_US = new Regex(#"^\d{5}$");
Regex rx_CA = new Regex(#"^[A-Za-z]\d[A-Za-z][ -]?\d[A-Za-z]\d$");
if (rx_US.IsMatch(zip))
{
queryFilter = queryFilter + " and Country eq 'US'";
}
else if (rx_CA.IsMatch(zip))
{
//the table search is case sensitive - test for common errors
zip = zip.ToUpper(); //make all upper case
Regex rx_CA1 = new Regex(#"^[A-Z]\d[A-Z]-\d[A-Z]\d$"); //dash
Regex rx_CA2 = new Regex(#"^[A-Z]\d[A-Z]\d[A-Z]\d$"); //no space
if (rx_CA1.IsMatch(zip)){zip = zip.Replace("-", " ");}
if (rx_CA2.IsMatch(zip)){zip = zip.Insert (3, " ");}
queryFilter = "$single&$filter=RowKey eq '" + zip + "'" + " and Country eq 'CA'";
}
string queryUrl = apiBaseUrl + queryFilter + authorization;
try
{
var httpWebRequest = WebRequest.Create(queryUrl);
httpWebRequest.ContentType = "application/json";
httpWebRequest.Headers.Add("Accept","application/json"); //if this is omitted you will get xml format
httpWebRequest.Method = "GET";
var httpResponse = await httpWebRequest.GetResponseAsync();
using(var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var responseText = streamReader.ReadToEnd();
apiResponse = responseText;
log.LogInformation("Full Table Response = " + responseText);
}
int i = 0;
while (httpResponse.Headers["x-ms-continuation-NextPartitionKey"] != null && apiResponse.Length < 105)
{
//if response is > 105 then it found something - don't keep looking
//if there are continuation tokens then keep querying until you find something
var partitionToken = httpResponse.Headers["x-ms-continuation-NextPartitionKey"];
var rowToken = httpResponse.Headers["x-ms-continuation-NextRowKey"];
var continuationUrl = "NextPartitionKey="+partitionToken+"&NextRowKey="+rowToken+"&";
queryUrl = apiBaseUrl + continuationUrl + queryFilter + authorization;
log.LogInformation("begin new httpRequest...");
httpWebRequest = WebRequest.Create(queryUrl);
httpWebRequest.ContentType = "application/json";
httpWebRequest.Headers.Add("Accept","application/json");
httpWebRequest.Method = "GET";
httpResponse = await httpWebRequest.GetResponseAsync();
using(var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var responseText = streamReader.ReadToEnd();
apiResponse = responseText;
log.LogInformation("Full Table Response = " + responseText);
}
i++;
log.LogInformation("loop # "+ i + " - url = " + queryUrl + " Response = "+ apiResponse);
}
if(apiResponse.Length > 105)
{
//strip out extra data
apiResponse = apiResponse.Remove(1,101);
apiResponse = apiResponse.Remove(apiResponse.Length - 2,2);
}
else
{
apiResponse = "No data found for zip = " + zip + " - Ensure you have proper format and case";
}
}
catch (Exception ex)
{
apiResponse = "error: " + ex.Message;
}
log.LogInformation("ZipPostal function completed and returned "+ apiResponse);
return apiResponse;
}

How to escape characters in Azure Table queries?

I want to query for rows where a column named message starts with: metric="foo"
I tried encoding = and " with percentage and hex codes but it did not work.
Microsoft documentations says special characters must be encoded but does not tell how: https://learn.microsoft.com/en-us/rest/api/storageservices/querying-tables-and-entities#query-string-encoding
What should the query look like when value being compared contains special characters?
If you're using azure sdk, then the sdk already did the tricky for you.
In my test, I'm using the latest azure table storage sdk Microsoft.Azure.Cosmos.Table, version 1.0.4.
The test code:
static void Main(string[] args)
{
string connstr = "xxxx";
var storageAccount = CloudStorageAccount.Parse(connstr);
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("myCustomer123");
TableQuery<CustomerEntity> query = new TableQuery<CustomerEntity>();
string myfilter = TableQuery.CombineFilters(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, "ivan"),
TableOperators.And,
//for metric="foo", like below.
TableQuery.GenerateFilterCondition("PhoneNumber", QueryComparisons.Equal, "metric=\"foo\"")
);
query.FilterString = myfilter;
var items = table.ExecuteQuery(query);
foreach (var item in items)
{
Console.WriteLine(item.RowKey);
Console.WriteLine(item.PhoneNumber);
}
Console.WriteLine("*****end******");
Console.ReadLine();
}
Test result:
If you want to use the parameter to filter results, you can use ?$filter=<your parameter>%20eq%20'<vaule>'. For example
var date = DateTime.Now.ToUniversalTime().AddYears(1).ToString("R");
var CanonicalizedResource = "/" + StorageAccountName + "/people";
var StringToSign = date + "\n" + CanonicalizedResource;
// List the containers in a storage account.
// ListContainersAsyncREST(StorageAccountName, StorageAccountKey, CancellationToken.None).GetAwaiter().GetResult();
var hmacsha = new HMACSHA256();
hmacsha.Key = Convert.FromBase64String(StorageAccountKey);
var sig= hmacsha.ComputeHash(UTF8Encoding.UTF8.GetBytes(StringToSign));
var sig1 = Convert.ToBase64String(sig);
Console.WriteLine(sig1);
String uri = "https://jimtestperfdiag516.table.core.windows.net/people" + "?$filter=PartitionKey%20eq%20'Jim'";
HttpClient client = new HttpClient();
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, uri);
httpRequestMessage.Headers.Add("x-ms-date", date);
var str = "SharedKeyLite " + StorageAccountName + ":" + sig1;
httpRequestMessage.Headers.TryAddWithoutValidation("Authorization", str);
httpRequestMessage.Headers.Add("x-ms-version", "2017-04-17");
httpRequestMessage.Headers.Add("Accept", "application/json;odata=fullmetadata");
var results = client.SendAsync(httpRequestMessage).Result;
var response = results.Content.ReadAsStringAsync().Result;
var objs = JsonConvert.DeserializeObject(response);
Console.WriteLine(objs);

Defining log analytic data sources via C#

How can I add the Windows Performance Counters shown in the screenshot below via a C# application?
I found out this can be done via the OperationalInsightsManagementClient. The code below will add a metric.
void AddWorkspaceDatasources(string resourceGroupName, string objectName, string counterName)
{
var client = new OperationalInsightsManagementClient(GetCredentials()) {SubscriptionId = subscriptionId};
var existing = client.DataSources.ListByWorkspace(
new ODataQuery<DataSourceFilter> {Filter = "kind eq 'WindowsPerformanceCounter'"},
resourceGroupName,
resourceGroupName);
if (!existing.Any(c => (c.Properties as JObject)["objectName"].ToString() == objectName && (c.Properties as JObject)["counterName"].ToString() == counterName))
{
var properties = new JObject();
properties["counterName"] = counterName;
properties["instanceName"] = "*";
properties["intervalSeconds"] = 10;
properties["objectName"] = objectName;
properties["collectorType"] = "Default";
client.DataSources.CreateOrUpdate(
resourceGroupName,
resourceGroupName,
Regex.Replace(objectName, "[^a-zA-Z0-9]", "") + Regex.Replace(counterName, "[^a-zA-Z0-9]", ""),
new DataSource
{
Kind = "WindowsPerformanceCounter",
Properties = properties
});
}
}

Azure WADPerformanceCountertable query returning all records while querying for specific interval

How to get records for specific interval from Azure WADPerformanceCounters table using Java API?
Tried following code, but its giving all records in table. It seems timestamp based filters are not working. Tried with PartitionKey, Timestamp, EventTick and TIMESTAMP column filtering but its same for all.
public static void main(String arg[]){
try
{
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudTableClient tableClient = storageAccount.createCloudTableClient();
CloudTable cloudTable = tableClient.getTableReference("WADPerformanceCountersTable");
Long currTime = System.currentTimeMillis();
Date currentDate = new Date(currTime);
Date endTime = getFormattedTimestamp(currentDate);
System.out.println("endTime:" + endTime);
// calculation of start Time to DB format in UTC
long offsetInMilliseconds = 1000 * 60 * 2;
Date startTime = getFormattedTimestamp(new Date(currentDate
.getTime()
- offsetInMilliseconds));
System.out.println("startTime:" + startTime);
long startPartitionKey = 621355968000000000L + startTime
.getTime() * 10000;
long endPartitionKey = 621355968000000000L + endTime.getTime() * 10000;
//Query using PartitionKey
TableQuery< PerfTableEntity > SQL = TableQuery.from(PerfTableEntity.class).where(
"PartitionKey ge '0" + startPartitionKey + "'").where(
"PartitionKey le '0" + endPartitionKey + "'").where(
"DeploymentId eq '<deplymentid>'").where(
"RoleInstance eq 'WebRole1_IN_0'").where(
"CounterName eq '\\Memory\\Page Faults/sec' or CounterName eq '\\Memory\\Page Reads/sec'");
for (PerfTableEntity pd : cloudTable.execute(SQL)) {
System.out.println("\ncounterName = " +pd.getCounterName() + "= " + pd.getCounterValue() + "||" + pd.getTimestamp());
}
}catch (Exception e){
// Output the stack trace.
e.printStackTrace();
}
}//main
private static Date getFormattedTimestamp(Date date) {
try {
SimpleDateFormat df = new SimpleDateFormat(
"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
df.setTimeZone(TimeZone.getTimeZone("UTC"));
String datestr = df.format(date);
return df.parse(datestr);
} catch (Exception e) {
return null;
}
}
Using stringBuilder to append 0 to PartitionKey resolved the issue.

Twitter request token returns 401 unauthorized. I cant find what I did wrong

I need to do twitter sign in for mobile apps. Thus far I have tried a lot of libraries including tweetsharp and linq2twitter. But I am getting nowhere. With all this trouble, I resorted to webclient calls, but I am once again stuck. When I make the call, I get the 401 Unauthorized error. I include my code here. Can you guys see anything wrong at all ? I am really stuck with this.
string oauth_signature_method = "HMAC-SHA1";
TimeSpan ts = DateTime.UtcNow - new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc);
string oauth_timestamp = Convert.ToInt64(ts.TotalSeconds).ToString();
string oauth_version = "1.0";
string oauth_consumer_key = "iD1232134AQ2Pb6Q";
string oauth_nonce = Convert.ToBase64String(new ASCIIEncoding().GetBytes(DateTime.Now.Ticks.ToString()));
SortedDictionary<string, string> sd = new SortedDictionary<string, string>();
sd.Add("oauth_version", oauth_version);
sd.Add("oauth_consumer_key", oauth_consumer_key);
sd.Add("oauth_nonce", oauth_nonce);
sd.Add("oauth_signature_method", oauth_signature_method);
sd.Add("oauth_timestamp", oauth_timestamp);
UrlEntity callback = new UrlEntity();
callback.Url = #"//tweeter";
string encodedCallbackUrl = HttpUtility.UrlEncode(callback.Url);
sd.Add("oauth_callback",encodedCallbackUrl);
WebClient wc = new WebClient();
wc.Headers.Add("User-Agent: randomAgent HTTP Client");
wc.Headers.Add("Host: api.twitter.com");
wc.Headers.Add(#"Accept: */*");
UrlEntity url = new UrlEntity();
url.Url = #"https://api.twitter.com/oauth/request_token";
string signature = CreateSignature(url, sd);
sd.Add("oauth_signature",signature);
string dataValues = "";
foreach (KeyValuePair<string, string> pair in sd)
{
dataValues += pair.Key + "=''" + pair.Value + "'',";
}
dataValues = dataValues.Substring(0, dataValues.Length - 1); // cuts off the last,
string headerVal = " Oauth " + dataValues;
wc.Headers.Add("Authorization",headerVal);
wc.UploadString(#"https://api.twitter.com/oauth/request_token", "");
wc.DownloadStringCompleted += WcOnDownloadStringCompleted;
Below is the code to make the signature.
public static string CreateSignature(UrlEntity url, SortedDictionary<string, string> sd)
{
string parameterString = "";
SortedDictionary<string, string> sd2 = new SortedDictionary<string, string>();
foreach (KeyValuePair<string, string> parameter in sd)
{
string encodedKey = HttpUtility.UrlEncode(parameter.Key);
string encodedvalue = HttpUtility.UrlEncode(parameter.Value);
sd2.Add(encodedKey,encodedvalue);
}
foreach (KeyValuePair<string, string> pair in sd2)
{
parameterString += pair.Key + "=" + pair.Value + "&";
}
parameterString = parameterString.Substring(0, parameterString.Length - 1); // cuts off the last &
string signatureString = "POST&";
var encodedUrl = HttpUtility.UrlEncode(url.Url); // percent encode URL
signatureString += encodedUrl + "&";
signatureString += HttpUtility.UrlEncode(parameterString); // percent encode entire parameter string
string consumerSecret = "supercorectConsumersecretfromtwitterwebsite";
string signingKey = HttpUtility.UrlEncode(consumerSecret) + "&";
HMACSHA1 hasher = new HMACSHA1(new ASCIIEncoding().GetBytes(signingKey));
string signatureStringFinal = Convert.ToBase64String(hasher.ComputeHash(new ASCIIEncoding().GetBytes(signatureString)));
return signatureStringFinal;
}
It is a bit messy, but I am stuck. The following links might help.
https://dev.twitter.com/docs/auth/implementing-sign-twitter
https://dev.twitter.com/docs/api/1/post/oauth/request_token
https://dev.twitter.com/docs/auth/creating-signature
EDIT:
I have taken a look at the request using fiddler.
The response message has this in it...
"Failed to validate oauth signature and token"
Which likely means the oauth signature is wrong. But I have looked and I have not seen anything.
I am not a .net expert but am setting up something similar for PHP and note that the Oauth attributes must be in alphabetical order. From a quick look the examples here don't appear to be. See https://dev.twitter.com/docs/auth/creating-signature.

Resources