Kusto Query from c# - azure

I want to retrieve data from Kusto DB from c# app can any one help me on this.
I have knowledge on writing the Kusto queries but I need some help on pulling data from Azure Kusto DB hosted in Azure.
I tried the following code but it's not working:
var client = Kusto.Data.Net.Client.KustoClientFactory.CreateCslQueryProvider("https://help.kusto.windows.net/Samples;Fed=true");
var reader = client.ExecuteQuery("MyTable | count");
// Read the first row from reader -- it's 0'th column is the count of records in MyTable
// Don't forget to dispose of reader when done.

Could you please elaborate what's not working (what is the error message you're getting) with the code above?
In addition, a full (though simple) example can be found below:
// This sample illustrates how to query Kusto using the Kusto.Data .NET library.
//
// For the purpose of demonstration, the query being sent retrieves multiple result sets.
//
// The program should execute in an interactive context (so that on first run the user
// will get asked to sign in to Azure AD to access the Kusto service).
class Program
{
const string Cluster = "https://help.kusto.windows.net";
const string Database = "Samples";
static void Main()
{
// The query provider is the main interface to use when querying Kusto.
// It is recommended that the provider be created once for a specific target database,
// and then be reused many times (potentially across threads) until it is disposed-of.
var kcsb = new KustoConnectionStringBuilder(Cluster, Database)
.WithAadUserPromptAuthentication();
using (var queryProvider = KustoClientFactory.CreateCslQueryProvider(kcsb))
{
// The query -- Note that for demonstration purposes, we send a query that asks for two different
// result sets (HowManyRecords and SampleRecords).
var query = "StormEvents | count | as HowManyRecords; StormEvents | limit 10 | project StartTime, EventType, State | as SampleRecords";
// It is strongly recommended that each request has its own unique
// request identifier. This is mandatory for some scenarios (such as cancelling queries)
// and will make troubleshooting easier in others.
var clientRequestProperties = new ClientRequestProperties() { ClientRequestId = Guid.NewGuid().ToString() };
using (var reader = queryProvider.ExecuteQuery(query, clientRequestProperties))
{
// Read HowManyRecords
while (reader.Read())
{
var howManyRecords = reader.GetInt64(0);
Console.WriteLine($"There are {howManyRecords} records in the table");
}
// Move on to the next result set, SampleRecords
reader.NextResult();
Console.WriteLine();
while (reader.Read())
{
// Important note: For demonstration purposes we show how to read the data
// using the "bare bones" IDataReader interface. In a production environment
// one would normally use some ORM library to automatically map the data from
// IDataReader into a strongly-typed record type (e.g. Dapper.Net, AutoMapper, etc.)
DateTime time = reader.GetDateTime(0);
string type = reader.GetString(1);
string state = reader.GetString(2);
Console.WriteLine("{0}\t{1,-20}\t{2}", time, type, state);
}
}
}
}
}

Related

Cosmos DB: How to detect request charges with LINQ queries

In Cosmos DB v3, I'm getting an IOrderedQueryable<T> using GetItemLinqQueryable<T>. This allows me to write custom queries. The problem is I'd like to track request charges whenever a query is materialized. How can this be accomplished?
When I execute methods like ReadItemAsyncand ExecuteStoredProcedureAsync, the returned object has a RequestCharge property, but I need to detect charges with linq queries.
You can use the extension method ToFeedIterator on your IOrderedQueryable.
using Microsoft.Azure.Cosmos.Linq;
var query = container.GetItemLinqQueryable<MyClass>()
.Where(c => true)
.ToFeedIterator();
while (query.HasMoreResults)
{
var response = await query.ReadNextAsync();
Console.WriteLine(response.RequestCharge);
foreach (var myClassInstance in response)
{
// do stuff
}
}
edit: if you need count or any aggregate function:
var query = container.GetItemLinqQueryable<MyClass>()
.Where(c => true);
Response<int> x = await query.CountAsync();
Console.WriteLine(x.RequestCharge);
int count = x; // Autoboxing
You can find the full list of available extension functions on GitHub.

Data From REST API In Azure

I have implemented REST API calls using a standalone c# console application. The API returns JSON which i'm deserializing and then storing it in the database.
Now i want to implement the entire logic in Azure platform so that it can invoked by passing start date and an end date and store location (it should run for three location) Below is the code:
static void Main()
{
MakeInventoryRequest();
}
static async void MakeInventoryRequest()
{
using (var client = new HttpClient())
{
var queryString = HttpUtility.ParseQueryString(string.Empty);
// Request headers
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "5051fx6yyy124hhfyuscf34f57ce9");
// Request parameters
queryString["query.locationNumbers"] = "4638";
queryString["availableFromDate"] = "2019-01-01";
queryString["availableToDate"] = "2019-03-07";
var uri = "https://api-test.location.cloud/api/v1/inventory?" + queryString;
using (var request = new HttpRequestMessage(HttpMethod.Get, uri))
using (var response = await client.SendAsync(request))
{
var stream = await response.Content.ReadAsStreamAsync();
if (response.IsSuccessStatusCode == true)
{
List<Inventory> l1 = DeserializeJsonFromStream<List<Inventory>>(stream);
InsertInventoryRecords(l1);
}
if (response.IsSuccessStatusCode == false)
{
throw new Exception("Error Response Code: " + response.StatusCode.ToString() + "Content is: " + response.Content.ReadAsStringAsync().Result.ToString());
}
}
}
}
Please suggest the best possible design using Azure components
With the information in hand I think you have multiple options , you need to find out which works for you the best . You can use Cloud service to host the console app ( you will have to change it to worker role , Visual studio will help you to convert that ) . I am not sure about the load which you are expecting but you can always increase and decrease the instance and these can be deployed to different geographies .
I see that you are persisting the data , if you want to do that you can use many of the SQL offerings . For invoking the REST API you can also azure functions and ADF.
Please feel free to comment if you want any more details on the same.

How can I access the data set before the most recently posted ones in firebase

Currently I have a code for firebase cloud functions that is able to read the latest data set coming in using the onWrite function. However, I wish to be able to read the data set directly before the latest data set (as shown in the picture where my latest would for EG be LHYfA9GkpwOMS0OysUd and the previous data set would be LHYF9jlBmlQL5qD19-D). Is there a way to do this with firebase cloud functions? The code below shows how I get the latest data and edit it! I want to use the latest data's etotal value to subtract that from the previous data's etotal value.
edit: my data values wont be replacing the previous set so I cannot use 'previous'
exports.editData = functions.database.ref('/AllData/immaphoton/A/{id}').onWrite((change, context) => {
const afterData = change.after;
if (afterData.exists()) {
//console.log('hey');
const data = afterData.val();
// set of data to multiply by turns ratio
var actualEIn = (data.ein)*200;
var actualEOut = (data.eout)*200;
var totalE = (actualEIn - actualEOut);
var actualTotalPower = (data.tp)*200;
var ISOts = (data.ts);
// add timezone offset to milliseconds
var localTS = moment(Date.parse(data.ts) + (8*1000*60*60)).format('YYYY-MM-DD');
// need to change ts to suit each type of data
}
return admin.database().ref('/editedData/immaphoton/A').push({
ein: actualEIn,
eout: actualEOut,
etotal: totalE,
tp: actualTotalPower,
timestamp: ISOts,
localtime: localTS,
});
});
If you know the key of the new child node, you can read the item before it by combining orderByKey().limitToLast():
var key = context.params.id; // key of new/updated child
var ref = admin.database().ref('/editedData/immaphoton/A');
var query = ref.orderByKey().endAt(key).limitToLast(2);
query.once("child_added").then(function(snapshot) {
console.log(snapshot.key);
});
This code is not dependent on running in Cloud Functions, except for how it determines the value of key. If you can determine the key another way, this would run with any of the Firebase JavaScript SDKs.

How to query firebase realtime database in cloud code

I am using Firebase cloud code and firebase realtime database.
My database structure is:
-users
-userid32
-userid4734
-flag=true
-userid722
-flag=false
-userid324
I want to query only the users who's field 'flag' is 'true' .
What I am doing currently is going over all the users and checking one by one. But this is not efficient, because we have a lot of users in the database and it takes more than 10 seconds for the function to run:
const functions = require('firebase-functions');
const admin = require("firebase-admin");
admin.initializeApp(functions.config().firebase);
exports.test1 = functions.https.onRequest((request, response) => {
// Read Users from database
//
admin.database().ref('/users').once('value').then((snapshot) => {
var values = snapshot.val(),
current,
numOfRelevantUsers,
res = {}; // Result string
numOfRelevantUsers = 0;
// Traverse through all users to check whether the user is eligible to get discount.
for (val in values)
{
current = values[val]; // Assign current user to avoid values[val] calls.
// Do something with the user
}
...
});
Is there a more efficient way to make this query and get only the relevant records? (and not getting all of them and checking one by one?)
You'd use a Firebase Database query for that:
admin.database().ref('/users')
.orderByChild('flag').equalTo(true)
.once('value').then((snapshot) => {
const numOfRelevantUsers = snapshot.numChildren();
When you need to loop over child nodes, don't treat the resulting snapshot as an ordinary JSON object please. While that may work here, it will give unexpected results when you order on a value with an actual range. Instead use the built-in Snapshot.forEach() method:
snapshot.forEach(function(userSnapshot) {
console.log(userSnapshot.key, userSnapshot.val());
}
Note that all of this is fairly standard Firebase Database usage, so I recommend spending some extra time in the documentation for both the Web SDK and the Admin SDK for that.

Seeing lots of exception in collectionSelfLink

I'm seeing a lot of exceptions in the collectionSelfLink when making DocumentDb call -- see image below.
I'm able to connect to DocumentDb and read data but these exceptions concern me -- especially in something that's pretty straight forward like a collectionSelfLink.
Any idea what may be causing them and how to fix them?
Here's the function that's using the selfLink
public async Task<IEnumerable<T>> ReadQuery<T>(string dbName, string collectionId, SqlQuerySpec query)
{
// Prepare collection self link
// IMPORTANT: This is where I'm seeing those exceptions when I inspect the collectionLink. Otherwise, I'm NOT getting any errors.
var collectionLink = UriFactory.CreateDocumentCollectionUri(dbName, collectionId);
var result = _client.CreateDocumentQuery<T>(collectionLink, query, null);
_client.CreateDocumentQuery<T>(collectionLink);
return await result.QueryAsync();
}
And here's the QueryAsync() extension method
public async static Task<IEnumerable<T>> QueryAsync<T>(this IQueryable<T> query)
{
var docQuery = query.AsDocumentQuery();
var batches = new List<IEnumerable<T>>();
do
{
var batch = await docQuery.ExecuteNextAsync<T>();
batches.Add(batch);
}
while (docQuery.HasMoreResults);
var docs = batches.SelectMany(b => b);
return docs;
}
So SelfLink is an internal property that is set by DocumentDB. It cannot be set by the user and will only be populated on resources that have been returned from a call to the server.
The UriFactory code that you are using is construction a link that can be used to execute operations, but it is not a SelfLink.
If you are looking at a SelfLink property on a newly initialized DocumentCollection() object the SelfLink will be null as it has not been persisted on the server yet. This would explain all those errors in debug watch.

Resources