How can i get the last 100 records from the WADLogsTable Ordered by date ?
I tried to do it with this piece of code but it doesn't work
var query = (from entity in tsc.CreateQuery<LogsObject>("WADLogsTable")
where entity.PartitionKey.CompareTo(startTime.ToUniversalTime().Ticks.ToString("D19")) >= 0
orderby entity.EventTickCount descending
select entity);
Where tsc is the TableServiceContext.
I can get the records but i'm interested in the recents logs.
Thanks,
Windows Azure table storage doesn't support sorting, so entities always come back sorted by their PartitionKey+RowKey. But I suspect the log entries are already in reverse chronological order. Aren't they?
[EDIT] Apparently they're not. :-)
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ATCommon.DiagnosticConfig);
CloudTableClient cloudTableClient = storageAccount.CreateCloudTableClient();
TableServiceContext serviceContext = cloudTableClient.GetDataServiceContext();
IQueryable<WadLogEntity> traceLogsTable = serviceContext.CreateQuery<WadLogEntity>("WADLogsTable");
var selection = from row in traceLogsTable where row.PartitionKey.CompareTo("0" + DateTime.UtcNow.AddHours(hours).Ticks) >= 0 select row;
//var selection = from row in traceLogsTable where row.PartitionKey.CompareTo("0" + DateTime.UtcNow.AddMinutes(-5.0).Ticks) >= 0 select row;
CloudTableQuery<WadLogEntity> query = selection.AsTableServiceQuery<WadLogEntity>();
IEnumerable<WadLogEntity> output = query.Execute();
return output.OrderByDescending(s => s.Timestamp).Take(100).ToList();
Related
I have this autoquery implementation
var q = AutoQuery.CreateQuery(request, base.Request).SelectDistinct();
var results = Db.Select<ProductDto>(q);
return new QueryResponse<ProductDto>
{
Offset = q.Offset.GetValueOrDefault(0),
Total = (int)Db.Count(q),
Results = results
};
The request has some joins:
public class ProductSearchRequest : QueryDb<GardnerRecord, ProductDto>
, ILeftJoin<GardnerRecord, RecordToBicCode>, ILeftJoin<RecordToBicCode, GardnerBicCode>
{
}
The records gets returned correctly but the total is wrong. I can see 40,000 records in database but it tells me there is 90,000. There is multiple RecordToBicCode for each GardnerRecord so it's giving me the number of records multiplied by the number of RecordToBicCode.
How do I match the total to the number of GardnerRecord matching the query?
I am using PostgreSQL so need the count statement to be like
select count(distinct r.id) from gardner_record r etc...
Dores OrmLite have a way to do this?
I tried:
var q2 = q;
q2.SelectExpression = "select count(distinct \"gardner_record\".\"id\")";
q2.OrderByExpression = null;
var count = Db.Select<int>(q2);
But I get object reference not set error.
AutoQuery is returning the correct total count for your query of which has left joins so will naturally return more results then the original source table.
You can perform a Distinct count with:
Total = Db.Scalar<long>(q.Select(x => Sql.CountDistinct(x.Id));
I have the following query to get a set of documents with their page views
var activityQuery = ActivityInfoProvider.GetActivities()
.Columns($"COUNT(*) AS [{ColumnNames.PageViews}]", ColumnNames.ActivityValue)
.WhereEquals(ColumnNames.ActivityType, ActivityTypes.BlogPostView)
.GroupBy(ColumnNames.ActivityValue)
.ToString(true);
var query = DocumentHelper.GetDocuments(PageTypes.BlogPost)
.Columns(_blogCardColumns)
.Source(src => src.LeftJoin(new QuerySourceTable($"({activityQuery})", "A"), "V.DocumentGUID", "A.ActivityValue"))
.OrderByDescending(ColumnNames.BlogPostDate)
.Published(selectOnlyPublished)
.NestingLevel(1)
.Path(path);
This query works well, but it returns me a set of treenodes and as far as I can see, it doesn't return me the PageViews column.
Is there a way to do a query and the result set include the page views column (or is there a way to get it from the treenode element)?
I have tried the following with the results, but it just returns null:
var nodes = query.ToList()
nodes.First().GetValue("PageViews");
but this returns null, even though if I get the query sql and run it in sql management studio, it returns 28 for PageViews
Ok as it turns out .ToList() returns a treenode collection. If I change the last line to
var dataset = query.Result;
It returns a dataset with all the columns I requested in _blogCardsColumns and then I can access my pageviews by using
foreach (DataRow row in dataSet.Tables[0].Rows)
{
int pageviews = int.Parse(dataRow["PageViews"].ToString())
}
I am doing a query with this filter:
(PartitionKey eq 'A' or PartionKey eq 'B' or ...) and RowKey eq 'RK'
I realized that this kind of query with 20 to 100 PKs is taking 3 to 5 seconds. The total quantity of items on the table is not much (more or less 1 million)
I think is doing a partial scan query. I assumed that it would do several puntual queries, but it seems is not the case.
My other option is do independent parallel queries and then merge the results.
Is this a good option for 100 items?
I will not have problems with the network connections? (I increase them with ServicePointManager.DefaultConnectionLimit)
Note: Not all the pair PK/RK will retrieve a record.
My other option is do independent parallel queries and then merge the results.
It will save the query time on Azure Storage, but it will spend more time on query request and result response. I have a table with 160K entities. I wrote two sample code to test the total times of query the entities from one query and multi queries. Here are my test result.
Following are my sample code.
Query entities from one query.
int entitesCount = 20;
TableQuery<CustomerEntity> customerQuery = new TableQuery<CustomerEntity>();
string filter = "(";
for (int i = 0; i < entitesCount; i++)
{
filter = filter + "PartitionKey eq '" + i + "'";
if (i < entitesCount - 1)
{
filter = filter + " or ";
}
}
filter = filter + ") and RowKey eq '42'";
customerQuery.FilterString = filter;
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
var customers = table.ExecuteQuery(customerQuery);
Console.WriteLine(customers.Count().ToString());
stopWatch.Stop();
TimeSpan ts = stopWatch.Elapsed;
Console.WriteLine(ts.ToString());
Query entities from parallel multi queries.
ServicePointManager.DefaultConnectionLimit = 100;
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
int entitesCount = 20;
List<CustomerEntity> customers = new List<CustomerEntity>();
var result = Parallel.For(0, entitesCount, new Action<int>(i =>
{
TableQuery<CustomerEntity> customerQuery = new TableQuery<CustomerEntity>();
customerQuery.FilterString = "PartitionKey eq '" + i.ToString() + "' and RowKey eq '88'";
var cs = table.ExecuteQuery(customerQuery);
foreach (var c in cs)
{
customers.Add(c);
}
}));
while (!result.IsCompleted) { }
Console.WriteLine(customers.Count.ToString());
stopWatch.Stop();
TimeSpan ts = stopWatch.Elapsed;
Console.WriteLine(ts.ToString());
Azure Table Storage: Efficient way to query multiple PK-RK pairs
I suggest you test it on your side and determine which way is a good one.
I have a an Azure table which serves as an event log. I need the most efficient way to read the bottom of the table to retrieve the most recent entries.
What is the most efficient way of doing this?
First of all, I would really advice you to base your partition key on UTC ticks. You can do it in a way that all the antities are ordered from latest to oldest.
Then if you want to get lets say 100 latest logs you just call (lets say that query is IQueryable something from your favorite client - we use Lucifure Stash): query.Take(100);
If you want to fetch entities for certain period you write: query.Where(x => x.PartitionKey <= value); or something similar.
The "value" variable has to be constructed based on the way you construct the values for partition key.
Assuming you want to fetch the data for last 15 minutes, try this pseudo code:
DateTime toDateTime = DateTime.UtcNow;
DateTime fromDateTime = toDateTime.AddMinutes(-15);
string myPartitionKeyFrom = fromDateTime.ToString("yy-MM");
string myPartitionKeyTo = toDateTime.ToString("yy-MM");
string query = "";
if (myPartitionKeyFrom.Equals(myPartitionKeyTo))//In case both time periods fall in same month, then we can directly hit that partition.
{
query += "(PartitionKey eq '" + myPartitionKeyFrom + "') ";
}
else // Otherwise we would need to do a greater than and lesser than stuff.
{
query += "(PartitionKey ge '" + myPartitionKeyFrom + "' and PartitionKey le '" + myPartitionKeyTo + "') ";
}
query += "and (RowKey ge '" + fromDateTime.ToString() + "' and RowKey le '" + toDateTime.ToString() + "')";
If you want to fetch latest 'n' number of entries then you need to slightly modify your PartitionKey and RowKey value, So that latest entries will be pushed to the top of the table.
For this you need to compute both the keys using DateTime.MaxValue.Subtract(DateTime.UtcNow).Ticks; instead of DateTime.UtcNow.
Microsoft provides a SemanticLogging framework that has a specific sink to log to Azure Table.
If you look at the library code, it generates a partition key (in reverse order) based on a Datetime :
static string GeneratePartitionKeyReversed(DateTime dateTime)
{
dateTime = dateTime.AddMinutes(-1.0);
return GetTicksReversed(
new DateTime(dateTime.Year, dateTime.Month, dateTime.Day, dateTime.Hour, dateTime.Minute, 0));
}
static string GetTicksReversed(DateTime dateTime)
{
return (DateTime.MaxValue - dateTime.ToUniversalTime())
.Ticks.ToString("d19", (IFormatProvider)CultureInfo.InvariantCulture);
}
So you can implement the same logic in your application to build your partitionkey.
If you want to retrieve the logs for a specific date range, you can write a query that looks like that:
var minDate = GeneratePartitionKeyReversed(DateTime.UtcNow.AddHours(-2));
var maxDate = GeneratePartitionKeyReversed(DateTime.UtcNow.AddHours(-1));
// Get the cloud table
var cloudTable = GetCloudTable();
// Build the query
IQueryable<DynamicTableEntity> query = cloudTable.CreateQuery<DynamicTableEntity>();
// condition for max date
query = query.Where(a => string.Compare(a.PartitionKey, maxDate,
StringComparison.Ordinal) >= 0);
// condition for min date
query = query.Where(a => string.Compare(a.PartitionKey, minDate,
StringComparison.Ordinal) <= 0);3
I've a problem with LINQ. Basically a third party database that I need to connect to is using the now depreciated text field (I can't change this) and I need to execute a distinct clause in my linq on results that contain this field.
I don't want to do a ToList() before executing the Distinct() as that will result in thousands of records coming back from the database that I don't require and will annoy the client as they get charged for bandwidth usage. I only need the first 15 distinct records.
Anyway query is below:
var query = (from s in db.tSearches
join sc in db.tSearchIndexes on s.GUID equals sc.CPSGUID
join a in db.tAttributes on sc.AttributeGUID equals a.GUID
where s.Notes != null && a.Attribute == "Featured"
select new FeaturedVacancy
{
Id = s.GUID,
DateOpened = s.DateOpened,
Notes = s.Notes
});
return query.Distinct().OrderByDescending(x => x.DateOpened);
I know I can do a subquery to do the same thing as above (tSearches contains unique records) but I'd rather a more straightfoward solution if available as I need to change a number of similar queries throughout the code to get this working.
No answers on how to do this so I went with my first suggestion and retrieved the unique records first from tSearch then constructed a subquery with the non unique records and filtered the search results by this subquery. Answer below:
var query = (from s in db.tSearches
where s.DateClosed == null && s.ConfidentialNotes != null
orderby s.DateOpened descending
select new FeaturedVacancy
{
Id = s.GUID,
Notes = s.ConfidentialNotes
});
/* Now filter by our 'Featured' attribute */
var subQuery = from sc in db.tSearchIndexes
join a in db.tAttributes on sc.AttributeGUID equals a.GUID
where a.Attribute == "Featured"
select sc.CPSGUID;
query = query.Where(x => subQuery.Contains(x.Id));
return query;