Azure Table Storage: track retries - azure

I'm using the standard example from microsoft for inserting new entitys to a table.
Is there a way to track if a retry was performed?
Code:
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("people");
CustomerEntity customer1 = new CustomerEntity("Harp", "Walter");
TableOperation insertOperation = TableOperation.Insert(customer1);
table.Execute(insertOperation);
With TransientFaultHandlingFramework it was easy to do:
var retryPol = new RetryPolicy<SqlAzureTransientErrorDetectionStrategy>(retryStrategy);
retryPol.Retrying += (obj, eventArgs) =>
{
var msg = String.Format("Retrying, CurrentRetryCount = {0} , Delay = {1}, Exception = {2}", eventArgs.CurrentRetryCount, eventArgs.Delay, eventArgs.LastException.Message);
System.Diagnostics.Debug.WriteLine(msg);
};

You can use the Enterprise Library Transient Fault Handling Application Block as explained in Rob's answer:
var retryPol = new RetryPolicy<StorageTransientErrorDetectionStrategy>(retryStrategy);
retryPol.Retrying += (obj, eventArgs) =>
{
var msg = String.Format("Retrying, CurrentRetryCount = {0} , Delay = {1}, Exception = {2}", eventArgs.CurrentRetryCount, eventArgs.Delay, eventArgs.LastException.Message);
System.Diagnostics.Debug.WriteLine(msg);
};
var options = new TableRequestOptions { RetryPolicy = new RetryPolicies.NoRetry() };
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("people");
CustomerEntity customer1 = new CustomerEntity("Harp", "Walter");
TableOperation insertOperation = TableOperation.Insert(customer1);
retryPol.ExecuteAction<TableResult>(() => { return table.Execute(insertOperation, options); });
If you prefer to use exclusively the Windows Azure Storage Client Library, you can create a custom retry policy that raises events, like this:
public class EventExponentialRetry : IRetryPolicy
{
private static readonly TimeSpan DefaultClientBackoff = TimeSpan.FromSeconds(4.0);
private const int DefaultClientRetryCount = 3;
private TimeSpan deltaBackoff;
private int maximumAttempts;
private ExponentialRetry retry;
public event EventHandler<RetryEventArgs> RaiseRetryEvent;
public EventExponentialRetry()
{
Initialize(DefaultClientBackoff, DefaultClientRetryCount);
}
public EventExponentialRetry(TimeSpan deltaBackoff, int maxAttempts)
{
Initialize(deltaBackoff, maxAttempts);
}
private void Initialize(TimeSpan deltaBackoff, int maxAttempts)
{
this.deltaBackoff = deltaBackoff;
this.maximumAttempts = maxAttempts;
retry = new ExponentialRetry(this.deltaBackoff, this.maximumAttempts);
}
public IRetryPolicy CreateInstance()
{
EventExponentialRetry newInstance = new EventExponentialRetry(this.deltaBackoff, this.maximumAttempts);
newInstance.RaiseRetryEvent = this.RaiseRetryEvent;
return newInstance;
}
public bool ShouldRetry(int currentRetryCount, int statusCode, Exception lastException, out TimeSpan retryInterval, OperationContext operationContext)
{
bool shouldRetry = retry.ShouldRetry(currentRetryCount, statusCode, lastException, out retryInterval, operationContext);
if (shouldRetry)
{
OnRaiseRetryEvent(new RetryEventArgs(currentRetryCount, statusCode, lastException, retryInterval, operationContext));
}
return shouldRetry;
}
protected virtual void OnRaiseRetryEvent(RetryEventArgs e)
{
// Make a temporary copy of the event to avoid possibility of
// a race condition if the last subscriber unsubscribes
// immediately after the null check and before the event is raised.
EventHandler<RetryEventArgs> handler = RaiseRetryEvent;
// Event will be null if there are no subscribers.
if (handler != null)
{
// Use the () operator to raise the event.
handler(this, e);
}
}
}
See the CustomAzureStorageRetryPolicySample GitHub project for a complete sample.

You can make use of the same Enterprise Transient Fault Handling Framework with your Table Storage (and its associated retry notifications).
To do so you disable the default Table Storage transient fault handling and use the enterprise storage fault handling instead.
var retryPol = new RetryPolicy<StorageTransientErrorDetectionStrategy>(retryStrategy);
retryPol.Retrying += (obj, eventArgs) =>
{
var msg = String.Format("Retrying, CurrentRetryCount = {0} , Delay = {1}, Exception = {2}", eventArgs.CurrentRetryCount, eventArgs.Delay, eventArgs.LastException.Message);
System.Diagnostics.Debug.WriteLine(msg);
};
var options = new TableRequestOptions { RetryPolicy = new RetryPolicies.NoRetry() };
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("people");
CustomerEntity customer1 = new CustomerEntity("Harp", "Walter");
TableOperation insertOperation = TableOperation.Insert(customer1);
retryPol.ExecuteAction<TableResult>(() => { return table.Execute(insertOperation, options); });

Related

Azure Function : When Function AutoScale, function called multiple times

In Azure Queue add single entry for queueitem but function called multiple times for same queueitem, not able to understand what is wrong, We have used Async/Await in function and code is also Async, below is Azure function settings,
below is function code,
public class CreateTempVMTempTablesFunction
{
private static Container container;
private static IProcessingFunctionService _processingFunctionService;
private static IAzureFunctionFailuresRepository _azureFunctionFailuresRepository;
private static ITrackLogEventsService _trackLogEventsService;
[FunctionName("CreateTempVMTempTablesFunction")]
public static async Task Run([QueueTrigger("%environment-plan%" + AzureFunctionConstants.CreateTempTableQueue, Connection = "AzureWebJobsStorage")]string myQueueItem, ILogger log)
{
string ErrorMessage = string.Empty;
var start = DateTime.Now;
FunctionStatusEnum IsSuccess = FunctionStatusEnum.Success;
log.LogInformation($"C# Queue trigger function processed: {myQueueItem} - {start}");
Guid tempid = new Guid(myQueueItem);
TempVM tempVM = new TempVM();
try
{
container = BusinessLogic.Helpers.SimpleInjectorWebApiInitializer.InitializeSingleton();;
_processingFunctionService = container.GetInstance<IProcessingFunctionService>();
_azureFunctionFailuresRepository = container.GetInstance<IAzureFunctionFailuresRepository>();
_trackLogEventsService = container.GetInstance<ITrackLogEventsService>();
tempVM = await _processingFunctionService.GetById(tempid);
if (tempVM != null)
{
FunctionStatusEnum IsAlreadyPerformed = await _azureFunctionFailuresRepository.GetAzureFunctionFailureStatus(AzureFunctionConstants.CreateTempVMTempTablesFunction, tempVM.Id);
if (IsAlreadyPerformed != FunctionStatusEnum.Success)
{
ResponseData response = await _processingFunctionService.CreateTempVMTempTables(tempid);
}
else
{
ErrorMessage = AzureFunctionConstants.FunctionAlreadyProcessed;
}
}
else
{
ErrorMessage = AzureFunctionConstants.TempVMNotFound;
}
}
catch (Exception ex)
{
IsSuccess = FunctionStatusEnum.Failed;
ErrorMessage = ex.ToString();
}
finally
{
AzureFunctionFailures azureFunctionFailures = new AzureFunctionFailures()
{
Id = Guid.NewGuid(),
FunctionName = AzureFunctionConstants.CreateTempVMTempTablesFunction,
QueueItem = myQueueItem,
ErrorMessage = ErrorMessage,
StartTime = start,
EndTime = DateTime.Now,
FailureTypeId = tempid,
FunctionStatus = IsSuccess,
ProcessTime = (DateTime.Now - start).TotalMilliseconds,
};
await _azureFunctionFailuresRepository.Add(azureFunctionFailures);
}
log.LogInformation($"End Time : {DateTime.Now} - QueueItem {myQueueItem}");
log.LogInformation($"Total Time : {DateTime.Now - start} - QueueItem {myQueueItem}");
}
}
I have check the code where added entry in queue, but only one entry is added for one queueitem.
This issue happened when added multiple entry in same queue (i.e. load testing where I have added 24 request only) for different queue item, when single queue is run then this is not happened, our functions are in App Service Plan with autoscaling
As mentioned in comments. Just set value of WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT as 1 in application settings of your function.

How to cancel a running trigger function within azure durable functions?

I am trying the fan-out, fan-in pattern. Here's my code
[FunctionName("af_cancellation")]
public static async Task<string> RunOrchestrator(
[OrchestrationTrigger] DurableOrchestrationContext context, ILogger log)
{
var taskList = new List<Task<string>>();
var tokenSource = new CancellationTokenSource();
taskList.Add(context.CallActivityAsync<string>("af_cancellation_Hello", new { ct = tokenSource.Token, city = "Tokyo" }));
taskList.Add(context.CallActivityAsync<string>("af_cancellation_Hello", new { ct = tokenSource.Token, city = "Seattle" }));
taskList.Add(context.CallActivityAsync<string>("af_cancellation_Hello", new { ct = tokenSource.Token, city = "London" }));
try
{
await Task.WhenAll(taskList);
}
catch (FunctionException)
{
log.LogError("trigger function failed");
tokenSource.Cancel();
}
string output = "";
foreach (var t in taskList)
{
output += t.Result;
}
return output;
}
I would like to cancel all the tasks in taskList if any of them throw an exception. What i am noticing is that await Task.WhenAll finishes all the tasks before moving forward.
Here's the sample trigger function
[FunctionName("af_cancellation_Hello")]
public static string SayHello([ActivityTrigger] DurableActivityContext context, ILogger log)
{
var data = context.GetInput<dynamic>();
var name = (string)data.city;
// unable to de-serialize cancellation token here but we'll ignore that.
var ct = JsonConvert.DeserializeObject<CancellationToken>(data.ct);
if (name != "London")
{
System.Threading.Thread.Sleep(1000 * 30);
}
else
{
System.Threading.Thread.Sleep(1000 * 10);
throw new FunctionException("don't like london");
}
log.LogInformation($"Saying hello to {name}.");
return $"Hello {name}!";
}
How can i achieve this?
According to this I think it's not possible. If you need to undo the job from the other activities that succeeded, you must take a look on the Saga Pattern and launch compensating activities.
more info:
https://microservices.io/patterns/data/saga.html
https://www.enterpriseintegrationpatterns.com/patterns/conversation/CompensatingAction.html

Azure Table Storage not saving all object properties

I have a problem with the Azure Table Storage. What I'm trying to achieve is saving the ChangeToken of the SharePoint list in order to use the webhooks properly.
Here is the code:
public class TablesHelper
{
private static readonly string TokenTableName = "TokenTable";
public static async Task<ListChangeToken> GetChangeTokenForListAsync(string listId)
{
var retrieveOperation = TableOperation.Retrieve<ListChangeToken>("Lists", listId, new List<string>() { "ChangeToken" });
var tableReference = await GetTableReferenceAsync(TokenTableName);
var tableResult = await tableReference.ExecuteAsync(retrieveOperation);
if(tableResult.Result != null)
{
return tableResult.Result as ListChangeToken;
}
return null;
}
public static async Task SaveChangeTokenForListAsync(ListChangeToken changeToken)
{
var insertOperation = TableOperation.Insert(changeToken);
var tableReference = await GetTableReferenceAsync(TokenTableName);
var result = await tableReference.ExecuteAsync(insertOperation);
}
private static async Task<CloudTable> GetTableReferenceAsync(string tableName)
{
var storageAccount = CloudStorageAccount.Parse(ConfigurationHelper.CloudStorage);
var tableClient = storageAccount.CreateCloudTableClient();
var reference = tableClient.GetTableReference(tableName);
await reference.CreateIfNotExistsAsync();
return reference;
}
}
The ListChangeToken class:
public class ListChangeToken : TableEntity
{
public ListChangeToken(string listId, string changeToken)
{
this.PartitionKey = "Lists";
this.RowKey = listId;
this.ChangeToken = changeToken;
}
public ListChangeToken() { }
public string ChangeToken { get; set;}
}
As per request, the function calling TablesHelper:
[FunctionName("EventHandler")]
public static async Task Run([QueueTrigger("events", Connection = "CloudStorage")]string myQueueItem, TraceWriter log)
{
var notificationGroup = Newtonsoft.Json.JsonConvert.DeserializeObject<NotificationGroup>(myQueueItem);
var contextHelper = new ContextHelper();
foreach (var notification in notificationGroup.Value)
{
UriBuilder uriBuilder = new UriBuilder();
uriBuilder.Scheme = "https";
uriBuilder.Host = ConfigurationHelper.TenantDomain;
uriBuilder.Path = notification.SiteUrl;
using (var ctx = contextHelper.GetAppOnlyContext(uriBuilder.ToString()))
{
//Read change token
var currentChangeToken = await TablesHelper.GetChangeTokenForListAsync(notification.Resource);
if(currentChangeToken == null)
{
log.Error($"No change token found for list {notification.Resource}. This is a NO GO. Please use the '/api/Setup' function.");
}
var listId = Guid.Parse(notification.Resource);
var changes = await CSOMHelper.GetListItemChangesAsync(ctx, listId, currentChangeToken.ChangeToken);
if(changes.Count > 0)
{
var lastChange = changes[changes.Count - 1];
//Save the last change token
var changeTokenValue = lastChange.ChangeToken.StringValue;
await TablesHelper.SaveChangeTokenForListAsync(new ListChangeToken(
notification.Resource,
changeTokenValue
));
await HandleChanges(ctx, changes);
}
}
}
log.Info($"C# Queue trigger function processed: {myQueueItem}");
}
The problem is that always, when using the "GetChangeTokenForListAsync" the Entity is received properly, but the .ChangeToken property is always null. It is also not visible when browsing with the Azure Storage Explorer. What am I doing wrong here?
The issue is related to the Azure Storage Emulator (V. 5.7.0.0). The same code works perfectly when working with the "live" Azure.

Azure Function with Event Hub out binding does not work

I am using Azure Function to send out message to multiple Event Hub outputs (from an input EH). My code is the following:
[FunctionName("Gateway")]
public static void Run([EventHubTrigger("%INPUT_HUB_NAME%", Connection = "iothubconnection", ConsumerGroup = "functiontest")]EventData[] eventHubMessage,
[EventHub("%OUT_HUB_NAME%", Connection = "eventhuboutput")]out EventData outputEventHubMessageHotPath,
[EventHub("%OUT_HUB_NAME%", Connection = "eventhuboutput2")]out EventData outputEventHubMessageColdPath,
TraceWriter log)
{
log.Info("**-- Start Azure Func -- **");
foreach (var ehMsg in eventHubMessage)
{
//section to build up the raw section
var rawMessageSection = GetPayload(ehMsg.GetBytes());
var deviceId = GetDeviceId(ehMsg);
log.Info($"Extracted deviceId: {deviceId}");
if (rawMessageSection.aggregates != null)
{
var message = CreateEHMessages("aggregates", rawMessageSection, deviceId, log);
outputEventHubMessageHotPath = message;
outputEventHubMessageColdPath = message;
}
if (rawMessageSection.events != null)
{
outputEventHubMessageColdPath = CreateEHMessages("events", rawMessageSection, deviceId, log);
}
if (rawMessageSection.ipis != null)
{
outputEventHubMessageColdPath = CreateEHMessages("ipis", rawMessageSection, deviceId, log);
}
if (rawMessageSection.errors != null)
{
outputEventHubMessageColdPath = CreateEHMessages("errors", rawMessageSection, deviceId, log);
}
if (rawMessageSection.batteries != null)
{
outputEventHubMessageColdPath = CreateEHMessages("batteries", rawMessageSection, deviceId, log);
}
//await Task.WhenAll(tasks);
}
outputEventHubMessageHotPath = outputEventHubMessageColdPath = null;
}
where:
public static EventData CreateEHMessages(string messageType, dynamic messageBatch, string deviceId, TraceWriter log)
{
var timezone = messageBatch.timezone;
var deviceInstanceId = messageBatch.deviceInstanceId;
int i = 0;
List<dynamic> result = new List<dynamic>();
foreach (var msg in messageBatch[messageType])
{
msg.deviceId = deviceId;
msg.timezone = timezone;
msg.deviceInstanceId = deviceInstanceId;
msg.type = messageType;
result.Add(msg);
i++;
}
var eventData = new EventData(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(result)));
eventData.PartitionKey = deviceInstanceId;
return eventData;
}
The problem is that this Function does NOT seem to publish to EventHub. I have tried to use different syntax of binding but i cannot get it to work.
I suspect is something to do with output bindings but I again tried many options.
Any idea?
You set out parameter multiple times, so all but the last assignments will be lost. But your last assignment is setting them to null, which essentially means you return no messages from your Function.
Have a look at ICollector instead.
Define your output parameters as Collectors:
[EventHub("%OUT_HUB_NAME%", Connection = "eventhuboutput")]
ICollector<EventData> outputEventHubMessageHotPath,
Then add every message to the Collectors, e.g.:
if (rawMessageSection.events != null)
{
outputEventHubMessageHotPath.Add(
CreateEHMessages("events", rawMessageSection, deviceId, log));
}

How to stop Azure Message Session Handler from timing out when listening for messages?

Below is my synchronous implementation of the IMessageSessionHandler interface. One problem I have is that when there are no messages in the queue the InitializeReceiverSync method will repeatedly run until the web job ungracefully times out in Azure, resulting in a failure. On the other hand if there are messages in the queue to be read, once they are read, the session closes and the program exits.
What is the best way to exit the method if there are no messages in the queue?
public void ReceiveMessagesFromAzure()
{
int i = 0;
while (i < 4)
{
var QueueName = ConfigurationManager.AppSettings["QueueName"];
var connectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"];
CancellationTokenSource cts = new CancellationTokenSource();
InitializeReceiverSync(connectionString, QueueName, cts.Token);
i++;
}
}
public void InitializeReceiverSync(string connectionString, string queueName, CancellationToken ct)
{
var receiverFactory = MessagingFactory.CreateFromConnectionString(connectionString);
ct.Register(() => receiverFactory.Close());
var client = receiverFactory.CreateQueueClient(queueName, ReceiveMode.PeekLock);
client.RegisterSessionHandler(
typeof(SessionHandler),
new SessionHandlerOptions
{
MessageWaitTimeout = TimeSpan.FromSeconds(5),
MaxConcurrentSessions = 1,
AutoComplete = false
});
}
class SessionHandler : IMessageSessionHandler
{
AzureRepository ar = new AzureRepository([queueName], [connectionString]);
void IMessageSessionHandler.OnMessage(MessageSession session, BrokeredMessage message)
{
ar.ProcessMessage(message);
}
void IMessageSessionHandler.OnCloseSession(MessageSession session)
{
Console.WriteLine("Session closed");
Environment.Exit(0);
}
void IMessageSessionHandler.OnSessionLost(Exception exception)
{
Console.WriteLine(exception.Message);
Console.WriteLine("Press any key to exit");
}
}

Resources