How to get query Statistics such as Time Taken to execute and other parameters in Bigquery using C#.
QueryRequest _r = new QueryRequest();
_r.Query = "SELECT Id, Name FROM [Sample.Test] LIMIT 1000";
QueryResponse _qr = _service.Jobs.Query(_r, "samplequery").Fetch();
List<string> _fieldNames = _qr.Schema.Fields.ToList().Select(x => x.Name).ToList() ;
List<Google.Apis.Bigquery.v2.Data.TableRow> _rows = _qr.Rows.ToList();
There is JobStatistics Class, but i am not getting job statistics from above query execution. Else If there is any other way to get statistics pls suggest.
Thanks
I got it.
Job _j = _service.Jobs.Get(_qr.JobReference.ProjectId, _qr.JobReference.JobId).Fetch();
JobStatistics _js = _j.Statistics;
this.StartTime = _js.StartTime;
this.EndTime = _js.EndTime;
this.BytesProcessed = _js.TotalBytesProcessed;
Related
In Django ORM, I want to filter records using in-operation.
Earlier when I was running the below query it was working fine:
EndTime = 2023-02-01 (Varchar)
week_list = ['2023-01-01','2023-01-21','2023-02-02','2023-02-12']
Query :
count = Stats.objects.filter(EndTime__in = week_list).values('EndTime ').annotate(dcount=Count('EndTime '))
From this query, I was getting the proper result.
But when I am trying the same with the below values it is not working.
EndTime = 2023-02-01 12:00:09 (DateTimeobject)
week_list = ['2023-01-01','2023-01-21','2023-02-02','2023-02-12']
count = Stats.objects.filter(EndTime__in = week_list).values('EndTime').annotate(dcount=Count('EndTime'))
In this case, it is not returning the output but I want the same data as the earlier query was giving with the DateTime object field.
can anyone please help ?? how can I achieve this?
I am trying to fetch the deviceScope of the deviceTwin. However, the deviceScope is not returned in the DeviceTwinDevice.
String iotHubConnectionString = "xxx";
DeviceTwin twinClient = DeviceTwin.createFromConnectionString(iotHubConnectionString);
SqlQuery sqlQuery = SqlQuery.createSqlQuery("*",SqlQuery.FromType.DEVICES,"capabilities.iotEdge=false",null);
Query twinQuery = twinClient.queryTwin(sqlQuery.getQuery());
DeviceTwinDevice d = twinClient.getNextDeviceTwin(twinQuery);
Thanks to https://github.com/Azure/azure-iot-sdk-java/issues/1157. Have to use registerManager
I have this autoquery implementation
var q = AutoQuery.CreateQuery(request, base.Request).SelectDistinct();
var results = Db.Select<ProductDto>(q);
return new QueryResponse<ProductDto>
{
Offset = q.Offset.GetValueOrDefault(0),
Total = (int)Db.Count(q),
Results = results
};
The request has some joins:
public class ProductSearchRequest : QueryDb<GardnerRecord, ProductDto>
, ILeftJoin<GardnerRecord, RecordToBicCode>, ILeftJoin<RecordToBicCode, GardnerBicCode>
{
}
The records gets returned correctly but the total is wrong. I can see 40,000 records in database but it tells me there is 90,000. There is multiple RecordToBicCode for each GardnerRecord so it's giving me the number of records multiplied by the number of RecordToBicCode.
How do I match the total to the number of GardnerRecord matching the query?
I am using PostgreSQL so need the count statement to be like
select count(distinct r.id) from gardner_record r etc...
Dores OrmLite have a way to do this?
I tried:
var q2 = q;
q2.SelectExpression = "select count(distinct \"gardner_record\".\"id\")";
q2.OrderByExpression = null;
var count = Db.Select<int>(q2);
But I get object reference not set error.
AutoQuery is returning the correct total count for your query of which has left joins so will naturally return more results then the original source table.
You can perform a Distinct count with:
Total = Db.Scalar<long>(q.Select(x => Sql.CountDistinct(x.Id));
I am doing a query with this filter:
(PartitionKey eq 'A' or PartionKey eq 'B' or ...) and RowKey eq 'RK'
I realized that this kind of query with 20 to 100 PKs is taking 3 to 5 seconds. The total quantity of items on the table is not much (more or less 1 million)
I think is doing a partial scan query. I assumed that it would do several puntual queries, but it seems is not the case.
My other option is do independent parallel queries and then merge the results.
Is this a good option for 100 items?
I will not have problems with the network connections? (I increase them with ServicePointManager.DefaultConnectionLimit)
Note: Not all the pair PK/RK will retrieve a record.
My other option is do independent parallel queries and then merge the results.
It will save the query time on Azure Storage, but it will spend more time on query request and result response. I have a table with 160K entities. I wrote two sample code to test the total times of query the entities from one query and multi queries. Here are my test result.
Following are my sample code.
Query entities from one query.
int entitesCount = 20;
TableQuery<CustomerEntity> customerQuery = new TableQuery<CustomerEntity>();
string filter = "(";
for (int i = 0; i < entitesCount; i++)
{
filter = filter + "PartitionKey eq '" + i + "'";
if (i < entitesCount - 1)
{
filter = filter + " or ";
}
}
filter = filter + ") and RowKey eq '42'";
customerQuery.FilterString = filter;
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
var customers = table.ExecuteQuery(customerQuery);
Console.WriteLine(customers.Count().ToString());
stopWatch.Stop();
TimeSpan ts = stopWatch.Elapsed;
Console.WriteLine(ts.ToString());
Query entities from parallel multi queries.
ServicePointManager.DefaultConnectionLimit = 100;
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
int entitesCount = 20;
List<CustomerEntity> customers = new List<CustomerEntity>();
var result = Parallel.For(0, entitesCount, new Action<int>(i =>
{
TableQuery<CustomerEntity> customerQuery = new TableQuery<CustomerEntity>();
customerQuery.FilterString = "PartitionKey eq '" + i.ToString() + "' and RowKey eq '88'";
var cs = table.ExecuteQuery(customerQuery);
foreach (var c in cs)
{
customers.Add(c);
}
}));
while (!result.IsCompleted) { }
Console.WriteLine(customers.Count.ToString());
stopWatch.Stop();
TimeSpan ts = stopWatch.Elapsed;
Console.WriteLine(ts.ToString());
Azure Table Storage: Efficient way to query multiple PK-RK pairs
I suggest you test it on your side and determine which way is a good one.
How can I select the count from a table and include a where clause to return a long? Ideally I would use db.Count instead of db.Select. I'm just not sure how to use db.Count and cannot find documentation on it.
long totalCount = 0;
using (IDbConnection db = dbFactory.OpenDbConnection())
{
totalCount = db.Count<Content>( ?? );
}
Console.WriteLine(totalCount);
You answered for you question in your comment ;) You should use Count extension method with expression parameter. Example below:
long amout = db.Count<Post>(x => x.Subject == "test");
OrmLite generates following sql:
SELECT Count(*) FROM POST WHERE (SUBJECT = 'test')