I am trying to understand caching in kusto. I know Azure Data Explorer gives caching OOB. However, if I set the below properties while making a call to ADX, how it behaves? I don't see any difference with and without these properties. It returns the data almost with-in the same response time. Also, assume if I make the call at 8 where the cache for that specific query will be set to 2 hours. When i call the same query at 9 using the code with same properties, will kusto ignore the cache properties because there is a cache set already or will it reset the cache to 2 hours again from 9?
clientRequestProperties.SetOption(
ClientRequestProperties.OptionQueryResultsCacheForceRefresh,
true
);
clientRequestProperties.SetOption(
ClientRequestProperties.OptionQueryResultsCacheMaxAge
2
);
there are different types of cache in Kusto - the one controlled by the properties you included in your questions is the query results cache, which isn't enabled by default.
note that when you set ClientRequestProperties.OptionQueryResultsCacheMaxAge, you need to pass a value of type timespan, e.g. "02:00:00" (or TimeSpan.FromHours(2)) for a 2 hour period (and not 2 as you showed in your question).
note that when you set ClientRequestProperties.OptionQueryResultsCacheForceRefresh to true, you are forcing a cache refresh for a specific query, and cached results won't be used.
the more common type referred to is the one controlled by a caching policy. This is enabled by default.
Related
I have
an endpoint in an Azure Function called "INSERT" that that inserts a
record in Table Storage using a batch operation.
an endpoint in a different Azure Function
called "GET" that gets a record in Table Storage.
If I insert an item and then immediately get that same item, then the item has not appeared yet!
If I delay by one second after saving, then I find the item.
If I delay by 10ms after saving, then I don't find the item.
I see the same symptom when updating an item. I set a date field on the item. If I get immediately after deleting, then some times the date field is not set yet.
Is this known behavior in Azure Table Storage? I know about ETags as described here but I cannot see how it applies to this issue.
I cannot easily provide a code sample because this is distributed among multiple functions and I think if I did put it in a simpler example, then there would be some mechanism that would see I am calling from the same ip or with the same client and manage to return the recently saved item.
As mentioned in the comments, Azure Table Storage is Strongly Consistent. Data is available to you as soon as it is written to Storage.
This is in contrast with Cosmos DB Table Storage where there are many consistency levels and data may not be immediately available for you to read after it is written depending on the consistent level set.
The issue was related to my code and queues running in the background.
I had shut down the Function that has queue triggers but to my surprise I found that the Function in my staging slot was picking items off the queue. That is why it made a difference whether I delay for a second or two.
And to the second part, why a date field is seemingly not set as fast as I get it. Well, it turns out I had filtered by columns, like this:
var operation = TableOperation.Retrieve<Entity>(partitionKey, id, new List<string> { "Content", "IsDeleted" });
And to make matters worse, the class "Entity" that I deserialize to, of course had default primitive values (such as "false") so it didn't look like they were not being set.
So the answer does not have much to do with the question, so in summary, for anyone finding this question because they are wondering the same thing:
The answer is YES - Table Storage is in fact strongly consistent and it doesn't matter whether you're 'very fast' or connect from another location.
I'm using an Azure function like a scheduled job, using the cron timer. At a specific time each morning it calls a stored procedure.
The function is now taking 4 mins to run a stored procedure that takes a few seconds to run in SSMS. This time is increasing despite efforts to successfully improve the speed of the stored procedure.
The function is not doing anything intensive.
using (SqlConnection conn = new SqlConnection(str))
{
conn.Open();
using (var cmd = new SqlCommand("Stored Proc Here", conn) { CommandType = CommandType.StoredProcedure, CommandTimeout = 600})
{
cmd.Parameters.Add("#Param1", SqlDbType.DateTime2).Value = DateTime.Today.AddDays(-30);
cmd.Parameters.Add("#Param2", SqlDbType.DateTime2).Value = DateTime.Today;
var result = cmd.ExecuteNonQuery();
}
}
I've checked and the database is not under load with another process when the stored procedure is running.
Is there anything I can do to speed up the Azure function? Or any approaches to finding out why it's so slow?
UPDATE.
I don't believe Azure functions is at fault, the issue seems to be with SQL Server.
I eventually ran the production SP and had a look at the execution plan. I noticed that the statistic were way out, for example a join expected the number of returned rows to be 20, but actual figure was closer to 800k.
The solution for my issue was to update the statistic on a specific table each week.
Regarding why that stats were out so much, well the client does a batch update each night and inserts several hundred thousand rows. I can only assume this affected the stats and it's cumulative, so it seems to get worse with time.
Please be careful adding with recompile hints. Often compilation is far more expensive than execution for a given simple query, meaning that you may not get decent perf for all apps with this approach.
There are different possible reasons for your experience. One common reason for this kind of scenario is that you got different query plans in the app vs ssms paths. This can happen for various reasons (I will summarize below). You can determine if you are getting different plans by using the query store (which records summary data about queries, plans, and runtime stats). Please review a summary of it here:
https://learn.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store?view=sql-server-2017
You need a recent ssms to get the ui, though you can use direct queries from any tds client.
Now for a summary of some possible reasons:
One possible reason for plan differences is set options. These are different environment variables for a query such as enabling ansi nulls on or off. Each different setting could change the plan choice and thus perf. Unfortunately the defaults for different language drivers differ (historical artifacts from when each was built - hard to change now without breaking apps). You can review the query store to see if there are different “context settings” (each unique combination of set options is a unique context settings in query store). Each different set implies different possible plans and thus potential perf changes.
The second major reason for plan changes like you explain in your post is parameter sniffing. Depending on the scope of compilation (example: inside a sproc vs as hoc query text) sql will sometimes look at the current parameter value during compilation to infer the frequency of the common value in future executions. Instead of ignoring the value and just using a default frequency, using a specific value can generate a plan that is optimal for a single value (or set of values) but potentially slower for values outside that set. You can see this in the query plan choice in the query store as well btw.
There are other possible reasons for performance differences beyond what I mentioned. Sometimes there are perf differences when running in mars mode vs not in the client. There may be differences in how you call the client drivers that impact perf beyond this.
I hope this gives you a few tools to debug possible reasons for the difference. Good luck!
For a project I worked on we ran into the same thing. Its not a function issue but a sql server issue. For us we were updating sprocs during development and it turns out that per execution plan, sql server will cache certain routes/indexes (layman explanation) and that gets out of sync for the new sproc.
We resolved it by specifying WITH (RECOMPILE) at the end of the sproc and the API call and SSMS had the same timings.
Once the system is settled, that statement can and should be removed.
Search on slow sproc fast ssms etc to find others who have run into this situation.
I have a 100GB Cosmos DB collection that I'm applying a new index policy to. After setting the new index policy, calls to the ReadDocumentCollectionAsync C# API always reports a value of IndexTransformationProgress with -1. I've checked the HTTP headers and the x-ms-documentdb-collection-index-transformation-progress header is also -1.
Running a query on a field with a range index, I can see the index gradually being rebuilt if I use an order by clause. I've tried using different consistencies and lazy loading, but progress is always reported as -1. I can't find what this means in the online documentation.
How can I determine when the index has been rebuilt, and get the progress of a rebuild in progress?
I'm using the NuGet package version 1.18.0.
I am from the CosmosDB engineering team.
We do have transformation progress now for a partitioned collections - this was released recently. This can be fetched by setting the header 'x-ms-documentdb-populatequotainfo' to True. This header, in addition to providing quota and usage information for a partitioned collection, is also used to piggyback the index transformation progress information for partitioned collections.
Editor: This answer is now out of date
I am from Cosmos DB engineering team. The IndexTransformationProgress header is not supported for partitioned collection currently. '-1' means unknown or not applicable. Sorry for the inconvenience. We are evaluating and planning to support it in near future.
I have a DocumentDB instance with about 4,000 documents. I just configured Azure Search to search and index it. This worked fine at first. Yesterday I updated the documents and indexed fields along with one UDF to index a complex field. Now the indexer is reporting that DocumentDB is reporting RequestRateTooLargeException. The docs on that error suggest throttling calls but it seems like Search would need to do that. Is there a workaround?
Azure Search code uses DocumentDb client SDK, which retries internally with the appropriate timeout when it encounters RequestRateTooLarge error. However, this only works if there're no other clients using the same DocumentDb collection concurrently. Check if you have other concurrent users of the collection; if so, consider adding capacity to the collection.
This could also happen because, due to some other issue with the data, DocumentDb indexer isn't able to make forward progress - then it will retry on the same data and may potentially encounter the same data problem again, akin a poison message. If you observe that a specific document (or a small number of documents) cause indexing problem, you can choose to ignore them. I'm pasting an excerpt from the documentation we're about to publish:
Tolerating occasional indexing failures
By default, an Azure Search indexer stops indexing as soon as even as single document fails to be indexed. Depending on your scenario, you can choose to tolerate some failures (for example, if you repeatedly re-index your entire datasource). Azure Search provides two indexer parameters to fine- tune this behavior:
maxFailedItems: The number of items that can fail indexing before an indexer execution is considered as failure. Default is 0.
maxFailedItemsPerBatch: The number of items that can fail indexing in a single batch before an indexer execution is considered
as failure. Default is 0.
You can change these values at any time by specifying one or both of these parameters when creating or updating your indexer:
PUT https://service.search.windows.net/indexers/myindexer?api-version=[api-version]
Content-Type: application/json
api-key: [admin key]
{
"dataSourceName" : "mydatasource",
"targetIndexName" : "myindex",
"parameters" : { "maxFailedItems" : 10, "maxFailedItemsPerBatch" : 5 }
}
Even if you choose to tolerate some failures, information about which documents failed is returned by the Get Indexer Status API.
I am caching couple of requests with following unique keys... I am using In-Memory cache.
urn:Product:/site.api.rest/product?pagenumber=0&pagesize=0¶ms=<Product><ProductID>3</ProductID></Product>
urn:Product:/site.api.rest/product?pagenumber=0&pagesize=0¶ms=<Product><ProductID>1</ProductID></Product>
urn:Product:/site.api.rest/product?pagenumber=0&pagesize=0¶ms=<Product><ProductID>5</ProductID></Product>
urn:Product:/site.api.rest/product?pagenumber=0&pagesize=0¶ms=<Product><ProductID>3</ProductID><Description>test</Description></Product>
...
...
Now, in create/update/delete I would like to remove specific cache (based upon what params passed in the request body, for example everything with 3.
In order to do that I would get singleNode from request params (for example: 3).
How can I?
Get all cache objects which matches <ProductID>3</ProductID> and remove it?
Also, what is the right approach to remove cache?
base.RequestContext.RemoveFromCache(base.Cache, cachekey);
or
CacheClient.Remove(keyname)?
There is a better approach, which is to use generational caching.
When you construct your cache key include a generation number e.g.
urn:Product:gen_1:/site.api.rest/product?pagenumber=0&pagesize=0¶ms=
(This number could be stored as a counter in your caching service.)
Then when you want to (pseudo) invalidate a large set of cached items, just increment the generation number.
Just be sure to set an expiry date on the cached items so that older generations are cleaned up over time.