tableclient.RetryPolicy Vs. TransientFaultHandling - c#-4.0

Both myself and a colleague have been tasked with finding connection-retry logic for Azure Table Storage. After some searching, I found this really cool Enterprise Library suite, which contains the Microsoft.Practices.TransientFaultHandling namespace.
Following a few code examples, I ended up creating an Incremental retry strategy, and wrapping one of our storage calls with the retryPolicy's ExecuteAction callback handler :
/// <inheritdoc />
public void SaveSetting(int userId, string bookId, string settingId, string itemId, JObject value)
{
// Define your retry strategy: retry 5 times, starting 1 second apart, adding 2 seconds to the interval each retry.
var retryStrategy = new Incremental(5, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(2));
var storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting(StorageConnectionStringName));
try
{
retryPolicy.ExecuteAction(() =>
{
var tableClient = storageAccount.CreateCloudTableClient();
var table = tableClient.GetTableReference(SettingsTableName);
table.CreateIfNotExists();
var entity = new Models.Azure.Setting
{
PartitionKey = GetPartitionKey(userId, bookId),
RowKey = GetRowKey(settingId, itemId),
UserId = userId,
BookId = bookId.ToLowerInvariant(),
SettingId = settingId.ToLowerInvariant(),
ItemId = itemId.ToLowerInvariant(),
Value = value.ToString(Formatting.None)
};
table.Execute(TableOperation.InsertOrReplace(entity));
});
}
catch (StorageException exception)
{
ExceptionHelpers.CheckForPropertyValueTooLargeMessage(exception);
throw;
}
}
}
Feeling awesome, I went to go show my colleague, and he smugly noted that we could do the same thing without having to include Enterprise Library, as the CloudTableClient object already has a setter for a retry policy. His code ended up looking like :
/// <inheritdoc />
public void SaveSetting(int userId, string bookId, string settingId, string itemId, JObject value)
{
var storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting(StorageConnectionStringName));
var tableClient = storageAccount.CreateCloudTableClient();
// set retry for the connection
tableClient.RetryPolicy = new ExponentialRetry(TimeSpan.FromSeconds(2), 3);
var table = tableClient.GetTableReference(SettingsTableName);
table.CreateIfNotExists();
var entity = new Models.Azure.Setting
{
PartitionKey = GetPartitionKey(userId, bookId),
RowKey = GetRowKey(settingId, itemId),
UserId = userId,
BookId = bookId.ToLowerInvariant(),
SettingId = settingId.ToLowerInvariant(),
ItemId = itemId.ToLowerInvariant(),
Value = value.ToString(Formatting.None)
};
try
{
table.Execute(TableOperation.InsertOrReplace(entity));
}
catch (StorageException exception)
{
ExceptionHelpers.CheckForPropertyValueTooLargeMessage(exception);
throw;
}
}
My Question :
Is there any major difference between these two approaches, aside from their implementations? They both seem to accomplish the same goal, but are there cases where it's better to use one over the other?

Functionally speaking both are the same - they both retries requests in case of transient errors. However there are few differences:
Retry policy handling in storage client library only handles retries for storage operations while transient fault handling retries not only handles storage operations but also retries SQL Azure, Service Bus and Cache operations in case of transient errors. So if you have a project where you're using more that storage but would like to have just one approach for handling transient errors, you may want to use transient fault handling application block.
One thing I liked about transient fault handling block is that you can intercept retry operations which you can't do with retry policy. For example, look at the code below:
var retryManager = EnterpriseLibraryContainer.Current.GetInstance<RetryManager>();
var retryPolicy = retryManager.GetRetryPolicy<StorageTransientErrorDetectionStrategy>(ConfigurationHelper.ReadFromServiceConfigFile(Constants.DefaultRetryStrategyForTableStorageOperationsKey));
retryPolicy.Retrying += (sender, args) =>
{
// Log details of the retry.
var message = string.Format(CultureInfo.InvariantCulture, TableOperationRetryTraceFormat, "TableStorageHelper::CreateTableIfNotExist", storageAccount.Credentials.AccountName,
tableName, args.CurrentRetryCount, args.Delay);
TraceHelper.TraceError(message, args.LastException);
};
try
{
var isTableCreated = retryPolicy.ExecuteAction(() =>
{
var table = storageAccount.CreateCloudTableClient().GetTableReference(tableName);
return table.CreateIfNotExists(requestOptions, operationContext);
});
return isTableCreated;
}
catch (Exception)
{
throw;
}
In the code example above, I could intercept retry operations and do something there if I want to. This is not possible with storage client library.
Having said all of this, it is generally recommended to go with storage client library retry policy for retrying storage operations as it is an integral part of the package and thus would be kept up to date with the latest changes to the library.

Related

About understanding partition lease expiration

I have an event hub with 4 partitions and 2 consumer groups. I have 2 webjobs that read the data using an EventProcessor. Both for a different consumer group
I have configured the event processors like this:
var host = new EventProcessorHost(
Guid.NewGuid().ToString(),
configurationManager.EventHubConfiguration.Path,
configurationManager.EventHubConfiguration.ConsumerGroupName,
configurationManager.EventHubConfiguration.ListenerConnectionString,
configurationManager.StorageConfiguration.ConnectionString)
{
PartitionManagerOptions = new PartitionManagerOptions
{
AcquireInterval = TimeSpan.FromSeconds(10),
RenewInterval = TimeSpan.FromSeconds(10),
LeaseInterval = TimeSpan.FromSeconds(30)
}
};
var options = EventProcessorOptions.DefaultOptions;
options.MaxBatchSize = 250;
await host.RegisterEventProcessorFactoryAsync(new PlanCareEventProcessorFactory(telemetryClient, configurationManager), options);
return host;
In my EventProcessor I keep track of the progress (some methods skipped to keep it short and readable):
internal class PlanCareEventProcessor : IEventProcessor
{
public Task OpenAsync(PartitionContext context)
{
namespaceManager = NamespaceManager.CreateFromConnectionString(configurationManager.EventHubConfiguration.ManagerConnectionString);
if (namespaceManager == null)
return;
var currentSeqNo = context.Lease.SequenceNumber;
var lastSeqNo = namespaceManager.GetEventHubPartition(context.EventHubPath, context.ConsumerGroupName, context.Lease.PartitionId).EndSequenceNumber;
var delta = lastSeqNo - currentSeqNo;
var msg = $"Last processed seqnr for partition {context.Lease.PartitionId}: {currentSeqNo} of {lastSeqNo} in consumergroup '{context.ConsumerGroupName}' (lag: {delta})";
telemetryClient.TrackTrace(new TraceTelemetry(msg, SeverityLevel.Information));
telemetryClient.TrackMetric(new MetricTelemetry($"Partition_Lag_{context.Lease.PartitionId}_{context.ConsumerGroupName}", delta));
return Task.CompletedTask;
}
public async Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> events)
{
progressCounter++;
...
await LogProgress(context);
}
private async Task LogProgress(PartitionContext context)
{
if (progressCounter >= 100)
{
await CheckPointAsync(context);
progressCounter = 0;
}
}
}
Now I noticed a difference in the webjobs when it comes to how often OpenAsync and CloseAsync are called. For one of the consumer groups this is about every half hour while for the other one it is several times a minute.
Since both webjobs use the same code and are running on the same app plan, what could be the reason for this?
It bothers me because checkpointing using await CheckPointAsync(context) is almost never done for one of the webjobs since it does not reach the threshold before the lease is gone.

Does using QueueClient.OnMessage inside an asynchronous method make sense?

I am calling an async method InsertOperation from an async method ConfigureConnectionString. Am I using the client.OnMessage call correctly? I want to process the messages in a queue asynchronously and then store them to the queue storage.
private static async void ConfigureConnectionString()
{
var connectionString =
"myconnstring";
var queueName = "myqueue";
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("test");
table.CreateIfNotExists();
Stopwatch sw = Stopwatch.StartNew();
await Task.Run(() => InsertOperation(connectionString, queueName, table));
sw.Stop();
Console.WriteLine("ElapsedTime " + sw.Elapsed.TotalMinutes + " minutes.");
}
private static async Task InsertOperation(string connectionString, string queueName, CloudTable table)
{
var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
client.OnMessage(message =>
{
var bodyJson = new StreamReader(message.GetBody<Stream>(), Encoding.UTF8).ReadToEnd();
var myMessage = JsonConvert.DeserializeObject<VerifyVariable>(bodyJson);
Console.WriteLine();
var VerifyVariableEntityObject = new VerifyVariableEntity()
{
ConsumerId = myMessage.ConsumerId,
Score = myMessage.Score,
PartitionKey = myMessage.ConsumerId,
RowKey = myMessage.Score
};
});
}
OnMessageAsync method provides async programming model, it enables us to process a message asynchronously.
client.OnMessageAsync(message =>
{
return Task.Factory.StartNew(() => ProcessMessage(message));
//you could perofrm table and queue storage in ProcessMessage method
}, options);
Without understanding the actual logic you want to achieve, it looks like you are not using OnMessage correctly.
OnMessage is a way to set up the queue client behavior for a long running client. It makes sense, for example, if you have a singleton instance in your application. In that case, you are specifing to the client how you want to handle any messages that are put in the queue.
In your example, however, you create the client, set up the OnMessage, and don't persist the client, so it effectively doesn't get anything accomplished.

How to set the EventProcessorHost to read events from now on (UTC)?

We are using the EventProcessorHost to receive events from Azure EventHubs. I've been unsuccessfully trying to configure it (through the EventProcessorOptions.InitialOffsetProvider) to read events from UTC now on but it always reads from the start of the feed. I am not saving checkpoints (and I even deleted the BLOB container created).
This is how I am setting it:
DateTime startDate = DateTime.UtcNow;
var epo = new EventProcessorOptions
{
MaxBatchSize = 100,
PrefetchCount = 100,
ReceiveTimeOut = TimeSpan.FromSeconds(120),
InitialOffsetProvider = (name) => startDate
};
Any guidance would be appreciated.
Think this changed in version 2.0.0 - Rajiv's code would now be:
var eventProcessorOptions = new EventProcessorOptions
{
InitialOffsetProvider = (partitionId) => EventPosition.FromEnqueuedTime(DateTime.UtcNow)
};
Here is an example block with fully qualified classnames:
private static async Task MainAsync(string[] args)
{
try{
Console.WriteLine("Registering EventProcessor...");
string AISEhConnectionStringEndpoint = Configuration["AISEhConnectionStringEndpoint"];
string AISEhConnectionStringSharedAccessKeyName = Configuration["AISEhConnectionStringSharedAccessKeyName"];
string AISEhConnectionStringSharedAccessKey = Configuration["AISEhConnectionStringSharedAccessKey"];
string EhConnectionString = $"Endpoint={AISEhConnectionStringEndpoint};SharedAccessKeyName={AISEhConnectionStringSharedAccessKeyName};SharedAccessKey={AISEhConnectionStringSharedAccessKey}";
string AISEhEntityPath = Configuration["AISEhEntityPath"];
string AISEhConsumerGroupName = Configuration["AISEhConsumerGroupName"];
string AISStorageContainerName = Configuration["AISStorageContainerName"];
string AISStorageAccountName = Configuration["AISStorageAccountName"];
string AISStorageAccountKey = Configuration["AISStorageAccountKey"];
string StorageConnectionString = string.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", AISStorageAccountName, AISStorageAccountKey);
var eventProcessorHost = new Microsoft.Azure.EventHubs.Processor.EventProcessorHost(
AISEhEntityPath,
AISEhConsumerGroupName,
EhConnectionString,
StorageConnectionString,
AISStorageContainerName);
var options = new Microsoft.Azure.EventHubs.Processor.EventProcessorOptions
{
InitialOffsetProvider = (partitionId) => Microsoft.Azure.EventHubs.EventPosition.FromEnqueuedTime(DateTime.UtcNow)
};
// Registers the Event Processor Host and starts receiving messages
await eventProcessorHost.RegisterEventProcessorAsync<GetEvents>(options);
Thread.Sleep(Timeout.Infinite);
// Disposes of the Event Processor Host
await eventProcessorHost.UnregisterEventProcessorAsync();
}
catch(Exception ex)
{
Console.WriteLine(ex.Message);
NLog.LogManager.GetCurrentClassLogger().Error(ex);
throw;
}
}
}
And here are my general settings with secrets/exact addresses obscured to help work things out, as I found working this out to be less pleasurable than extracting teeth:
"AISEhConnectionStringEndpoint": "sb://<my bus address>.servicebus.windows.net/",
"AISEhConnectionStringSharedAccessKeyName": "<my key name>",
"AISEhConnectionStringSharedAccessKey": "<yeah nah>",
"AISEhEntityPath": "<Event Hub entity path>",
"AISEhConsumerGroupName": "<consumer group name e.g $Default>",
"AISStorageContainerName": "<storage container name>",
"AISStorageAccountName": "<storage account name>",
"AISStorageAccountKey": "<yeah nah>",
I found that the checkpoint folder in the blob was still there and my app was considering this and ignoring the date I set in EventProcessorOptions. After I deleted the container it started to run as expected (taking in count the UTC date).
You can use the EventProcessorOptions class for this and provide an offset set to the desired time.
var eventProcessorOptions = new EventProcessorOptions
{
InitialOffsetProvider = (partitionId) => DateTime.UtcNow
};
You can then use any of the RegisterEventProcessAsync overloads that accepts eventProcessorOptions.

Servicestack RabbitMQ: Infinite loop fills up dead-letter-queue when RabbitMqProducer cannot redeclare temporary queue in RPC-pattern

When I declare a temporary reply queue to be exclusive (e.g. anonymous queue (exclusive=true, autodelete=true) in rpc-pattern), the response message cannot be posted to the specified reply queue (e.g. message.replyTo="amq.gen-Jg_tv8QYxtEQhq0tF30vAA") because RabbitMqProducer.PublishMessage() tries to redeclare the queue with different parameters (exclusive=false), which understandably results in an error.
Unfortunately, the erroneous call to channel.RegisterQueue(queueName) in RabbitMqProducer.PublishMessage() seems to nack the request message in the incoming queue so that, when ServiceStack.Messaging.MessageHandler.DefaultInExceptionHandler tries to acknowlege the request message (to remove it from the incoming queue), the message just stays on top of the incoming queue and gets processed all over again. This procedure repeats indefinitely and results in one dlq-message per iteration which slowly fills up the dlq.
I am wondering,
if ServiceStack handles the case, when ServiceStack.RabbitMq.RabbitMqProducer cannot declare the response queue, correctly
if ServiceStack.RabbitMq.RabbitMqProducer muss always declare the response queue before publishing the response
if it wouldn't be best to have some configuration flag to omit all exchange and queue declaration calls (outside of the first initialization). The RabbitMqProducer would just assume every queue/exchange to be properly set up and just publish the message.
(At the moment our client just declares its response queue to be exclusive=false and everything works fine. But I'd really like to use rabbitmq's built-in temporary queues.)
MQ-Client Code, requires simple "SayHello" service:
const string INQ_QUEUE_NAME = "mq:SayHello.inq";
const string EXCHANGE_NAME="mx.servicestack";
var factory = new ConnectionFactory() { HostName = "192.168.179.110" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
// Create temporary queue and setup bindings
// this works (because "mq:tmp:" stops RabbitMqProducer from redeclaring response queue)
string responseQueueName = "mq:tmp:SayHello_" + Guid.NewGuid().ToString() + ".inq";
channel.QueueDeclare(responseQueueName, false, false, true, null);
// this does NOT work (RabbitMqProducer tries to declare queue again => error):
//string responseQueueName = Guid.NewGuid().ToString() + ".inq";
//channel.QueueDeclare(responseQueueName, false, false, true, null);
// this does NOT work either (RabbitMqProducer tries to declare queue again => error)
//var responseQueueName = channel.QueueDeclare().QueueName;
// publish simple SayHello-Request to standard servicestack exchange ("mx.servicestack") with routing key "mq:SayHello.inq":
var props = channel.CreateBasicProperties();
props.ReplyTo = responseQueueName;
channel.BasicPublish(EXCHANGE_NAME, INQ_QUEUE_NAME, props, Encoding.UTF8.GetBytes("{\"ToName\": \"Chris\"}"));
// consume response from response queue
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume(responseQueueName, true, consumer);
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
// print result: should be "Hello, Chris!"
Console.WriteLine(Encoding.UTF8.GetString(ea.Body));
}
}
Everything seems to work fine when RabbitMqProducer does not try to declare the queues, like that:
public void PublishMessage(string exchange, string routingKey, IBasicProperties basicProperties, byte[] body)
{
const bool MustDeclareQueue = false; // new config parameter??
try
{
if (MustDeclareQueue && !Queues.Contains(routingKey))
{
Channel.RegisterQueueByName(routingKey);
Queues = new HashSet<string>(Queues) { routingKey };
}
Channel.BasicPublish(exchange, routingKey, basicProperties, body);
}
catch (OperationInterruptedException ex)
{
if (ex.Is404())
{
Channel.RegisterExchangeByName(exchange);
Channel.BasicPublish(exchange, routingKey, basicProperties, body);
}
throw;
}
}
The issue got adressed in servicestack's version v4.0.32 (fixed in this commit).
The RabbitMqProducer no longer tries to redeclare temporary queues and instead assumes that the reply queue already exist (which solves my problem.)
(The underlying cause of the infinite loop (wrong error handling while publishing response message) probably still exists.)
Edit: Example
The following basic mq-client (which does not use ServiceStackmq client and instead depends directly on rabbitmq's .net-library; it uses ServiceStack.Text for serialization though) can perform generic RPCs:
public class MqClient : IDisposable
{
ConnectionFactory factory = new ConnectionFactory()
{
HostName = "192.168.97.201",
UserName = "guest",
Password = "guest",
//VirtualHost = "test",
Port = AmqpTcpEndpoint.UseDefaultPort,
};
private IConnection connection;
private string exchangeName;
public MqClient(string defaultExchange)
{
this.exchangeName = defaultExchange;
this.connection = factory.CreateConnection();
}
public TResponse RpcCall<TResponse>(IReturn<TResponse> reqDto, string exchange = null)
{
using (var channel = connection.CreateModel())
{
string inq_queue_name = string.Format("mq:{0}.inq", reqDto.GetType().Name);
string responseQueueName = channel.QueueDeclare().QueueName;
var props = channel.CreateBasicProperties();
props.ReplyTo = responseQueueName;
var message = ServiceStack.Text.JsonSerializer.SerializeToString(reqDto);
channel.BasicPublish(exchange ?? this.exchangeName, inq_queue_name, props, UTF8Encoding.UTF8.GetBytes(message));
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume(responseQueueName, true, consumer);
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
//channel.BasicAck(ea.DeliveryTag, false);
string response = UTF8Encoding.UTF8.GetString(ea.Body);
string responseType = ea.BasicProperties.Type;
Console.WriteLine(" [x] New Message of Type '{1}' Received:{2}{0}", response, responseType, Environment.NewLine);
return ServiceStack.Text.JsonSerializer.DeserializeFromString<TResponse>(response);
}
}
~MqClient()
{
this.Dispose();
}
public void Dispose()
{
if (connection != null)
{
this.connection.Dispose();
this.connection = null;
}
}
}
Key points:
client declares anonymous queue (=with empty queue name) channel.QueueDeclare()
server generates queue and returns queue name (amq.gen*)
client adds queue name to message properties (props.ReplyTo = responseQueueName;)
ServiceStack automatically sends response to temporary queue
client picks up response and deserializes
It can be used like that:
using (var mqClient = new MqClient("mx.servicestack"))
{
var pingResponse = mqClient.RpcCall<PingResponse>(new Ping { });
}
Important: You've got to use servicestack version 4.0.32+.

Azure Table Storage Warning - WCF Data Services obsolete

After upgrading to the new storage API version 4.2, I'm getting the following warning that I'm calling obsolete methods on some of my segmented queries.
'Microsoft.WindowsAzure.Storage.Table.CloudTableClient.GetTableServiceContext()'
is obsolete: 'Support for accessing Windows Azure Tables via WCF Data
Services is now obsolete. It's recommended that you use the
Microsoft.WindowsAzure.Storage.Table namespace for working with
tables.'
So far I haven't been able to figure out how to achieve this on the new API, and no examples have been put out that I have been able to find. The legacy code still runs fine, but if the new API supports something better I'd love to check it out and get rid of this warning. Could someone point me in the right direction on how a segmented query like this would look using the new API?
Here is what my code currently looks like with the warning:
public AzureTablePage<T> GetPagedResults<T>(Expression<Func<T, bool>> whereCondition, string ContinuationToken, int PageSize, string TableName) {
TableContinuationToken token = GetToken(ContinuationToken);
var query = AzureTableService.CreateQuery<T>(TableName).Where(whereCondition).Take(PageSize).AsTableServiceQuery(AzureTableClient.GetTableServiceContext());
var results = query.ExecuteSegmented(token, new TableRequestOptions() { PayloadFormat = TablePayloadFormat.JsonNoMetadata });
if (results.ContinuationToken != null) {
return new AzureTablePage<T>() { Results = results.ToList(), HasMoreResults = true, ContinuationToken = string.Join("|", results.ContinuationToken.NextPartitionKey, results.ContinuationToken.NextRowKey) };
} else {
return new AzureTablePage<T>() { Results = results.ToList(), HasMoreResults = false };
}
}
public TableServiceContext AzureTableService {
get {
var context = AzureTableClient.GetTableServiceContext();
context.IgnoreResourceNotFoundException = true;
return context;
}
}
public CloudTableClient AzureTableClient {
get {
return mStorageAccount.CreateCloudTableClient();
}
}
Solution
For anyone with the same question, here is the updated code.
/* Add the following Using Statement */
using Microsoft.WindowsAzure.Storage.Table.Queryable;
public AzureTablePage<T> GetPagedResults<T>(Expression<Func<T, bool>> whereCondition, string ContinuationToken, int PageSize, string TableName) where T : class, ITableEntity, new() {
TableContinuationToken token = GetToken(ContinuationToken);
var query = AzureTableClient.GetTableReference(TableName).CreateQuery<T>().Where(whereCondition).Take(PageSize).AsTableQuery();
var results = query.ExecuteSegmented(token, new TableRequestOptions() { PayloadFormat = TablePayloadFormat.JsonNoMetadata });
if (results.ContinuationToken != null) {
return new AzureTablePage<T>() { Results = results.ToList(), HasMoreResults = true, ContinuationToken = string.Join("|", results.ContinuationToken.NextPartitionKey, results.ContinuationToken.NextRowKey) };
} else {
return new AzureTablePage<T>() { Results = results.ToList(), HasMoreResults = false };
}
}
Please see the Tables Deep Dive blog post that we published when we first introduced the new Table Service Layer. If you need LINQ support, please also see the Azure Storage Client Library 2.1 blog post.
We strongly recommend upgrading to Table Service Layer, because it is optimized for NoSQL scenarios and therefore provides much better performance.

Resources