Azure Table Storage offers a BatchOperation method. It returns a list of TableResults. From what I've seen, there is never a time where this return value will have mixed failures and successes (as a batch should be). I haven't been able to find documentation that says this is a fact though. If anyone has a handy link to this specific info let me know.
TableBatch operation is atomic, so there is no point to continue executing the batch operation after the first failure. There are 2 outcomes for a TableBatchOperation, either all operations succeed and the overall request succeeds or the request returns on the first failed operation, and the changes made by previous operations are rolled back.
The interesting thing here is that, you will get a StorageException if there is a failure in one of the operations in the batch and the index of the failed operation is embedded inside the StorageException object. Then if you want to, you can implement the logic to automatically remove that operation from the batch (and log) and resubmit the TableBatchOperation.
I have implemented a StorageException extension class which extracts the failed operation index and many other useful information from the StorageException object.
Feel free to use it:
https://www.nuget.org/packages/AzureStorageExceptionParser/
Related
I have the following setup:
Azure function in Python 3.6 is processing some entities utilizing
the TableService class from the Python Table API (new one for
CosmosDB) to check whether the currently processed entity is already
in a storage table. This is done by invoking
TableService.get_entity.
get_entity method throws exception each
time it does not find an entity with the given rowid (same as the
entity id) and partition key.
If no entity with this id is found then i call insert_entity to add it to the table.
With this i am trying to implement processing only of entities that haven't been processed before (not logged in the table).
I observe however consistent behaviour of the function simply freezing after exactly 10 exceptions, where it halts execution on the 10th invocation and does not continue processing for another minute or 2!
I even changed the implementation of instead doing a lookup first to simply call insert_entity and let it fail when a duplicate row key is added. Surprisingly enough the behaviour there is exactly the same - on the 10th duplicate invocation the execution freezes and continues after a while.
What is the cause of such behaviour? Is this some sort of protection mechanism on the storage account that kicks in? Or is it something to do with the Python client? To me it looks very much something by design.
I wasnt able to find any documentation or settings page on the portal for influencing such behaviour.
I am wondering of it is possible to implement such logic with using table storage? I don't seem to find it justifiable to spin up a SQL Azure db or Cosmos DB instance for such trivial functionality of checking whether an entity does not exist in a table.
Thanks
I am bulk indexing into Elasticsearch docs containing country shapes (files here), based on the cshapes dataset.
The geoshapes have a lot of points in "geometry":{"type":"MultiPolygon", and the bulk request takes a long time to complete (and sometimes does not complete, which is a separate and already reported problem).
Since the client times out (I use the official ES node.js), I would like to have a way to check what the status of the bulk request is, without having to use enormous timeout values.
What I would like is to have a status such as active/running, completed or aborted. I guess that just by querying the single doc in the batch would not tell me whether the request has been aborted.
Is this possible?
I'm not sure if this is exactly what you're looking for, but may be helpful. Whenever I'm curious about what my cluster is doing, I check out the tasks API.
The tasks API shows you all of the tasks that are currently running on your cluster. It will give you information about individual tasks, such as the task ID, start time, and running time. Here's the command:
curl -XGET http://localhost:9200/_tasks?group_by=parents | python -m json.tool
Elasticsearch doesn't provide a way to check the status of an ongoing Bulk request- documentation reference here.
First, check that your request succeeds with a smaller input, so you know there is no problem with the way you are making the request. Second, try dividing the data into smaller chunks and calling the Bulk API on them in parallel.
You can also try with a higher request_timeout value, but I guess that is something you don't want to do.
just a side note hint, of why your requests might take a lot of time (unless you are just indexing too many in a single bulk run). If you have configured your own precision for geo shapes, also make sure you are configuring distance_error_pct, otherwise no error is assumed, resulting in documents with a lot of terms that take a lot of time indexing.
I have a DocumentDB instance with about 4,000 documents. I just configured Azure Search to search and index it. This worked fine at first. Yesterday I updated the documents and indexed fields along with one UDF to index a complex field. Now the indexer is reporting that DocumentDB is reporting RequestRateTooLargeException. The docs on that error suggest throttling calls but it seems like Search would need to do that. Is there a workaround?
Azure Search code uses DocumentDb client SDK, which retries internally with the appropriate timeout when it encounters RequestRateTooLarge error. However, this only works if there're no other clients using the same DocumentDb collection concurrently. Check if you have other concurrent users of the collection; if so, consider adding capacity to the collection.
This could also happen because, due to some other issue with the data, DocumentDb indexer isn't able to make forward progress - then it will retry on the same data and may potentially encounter the same data problem again, akin a poison message. If you observe that a specific document (or a small number of documents) cause indexing problem, you can choose to ignore them. I'm pasting an excerpt from the documentation we're about to publish:
Tolerating occasional indexing failures
By default, an Azure Search indexer stops indexing as soon as even as single document fails to be indexed. Depending on your scenario, you can choose to tolerate some failures (for example, if you repeatedly re-index your entire datasource). Azure Search provides two indexer parameters to fine- tune this behavior:
maxFailedItems: The number of items that can fail indexing before an indexer execution is considered as failure. Default is 0.
maxFailedItemsPerBatch: The number of items that can fail indexing in a single batch before an indexer execution is considered
as failure. Default is 0.
You can change these values at any time by specifying one or both of these parameters when creating or updating your indexer:
PUT https://service.search.windows.net/indexers/myindexer?api-version=[api-version]
Content-Type: application/json
api-key: [admin key]
{
"dataSourceName" : "mydatasource",
"targetIndexName" : "myindex",
"parameters" : { "maxFailedItems" : 10, "maxFailedItemsPerBatch" : 5 }
}
Even if you choose to tolerate some failures, information about which documents failed is returned by the Get Indexer Status API.
When executing a batch, if one of the operation of TableBatchOperation fails:
Every operations in the batch are canceled
Every other operations that are valid are processed
The first valid operations in the queue are processed until one operation fails, and following ones are not processed
Answer is 1 - Even if one operation fails in the batch, the entire operation fails (or in other words rolls back). This is similar to performing transactions in a relational database. What's interesting is that you get an index of the failed entity in the response when this happens. Check this thread for more details: Azure CloudTable.ExecuteBatch(TableBatchOperation) throws a storageexception. How can I find which operation(s) caused the exception?
Official blog post: http://blogs.msdn.com/b/windowsazurestorage/archive/2012/11/06/windows-azure-storage-client-library-2-0-tables-deep-dive.aspx
TableBatchOperations, or Entity Group Transactions, are executed
atomically meaning that either all operations will succeed or if there
is an error caused by one of the individual operations the entire
batch will fail.
Is it possible to cancel an Azure table query?
We have cases where we are making a long running query (can take 30-60 seconds), but the object gets disposed and needs to abort the query before it completes.
We are using TableServicesContext, and ExecuteQuery (synchronously). We can consider async as well if the solution requires it.
First of all, I doubt a table service query may last longer than 30 seconds. Check out this documentation on Query Timeouts and Pagination.
Also, the Windows Azure Storage Services SLA guarantees that the maximum response time for a Table Service (which is for batch operation) is 30 seconds. And operations on single entities shall complete within 2 seconds.
If yet, you still having issues, your solution is to use BeginExecute method which will give you back an IAsyncResult object. You can have your own timer and call CancelRequest with the given IAsyncResult upon your own logic.
By now, if you followed all my links, you might have noticed that BeginExecute and CancelRequest are methods of DataServiceContext calss. That's why they are not complete in the documentation for TableSeriveContext. But since TableServiceContext inherits directly DataServiceContext, these methods are availabe in your TableServiceContext also.
You may also want to take a look at How to: Execute Asynchronous Data Service Queries
Hope this helps!