In Azure Queue add single entry for queueitem but function called multiple times for same queueitem, not able to understand what is wrong, We have used Async/Await in function and code is also Async, below is Azure function settings,
below is function code,
public class CreateTempVMTempTablesFunction
{
private static Container container;
private static IProcessingFunctionService _processingFunctionService;
private static IAzureFunctionFailuresRepository _azureFunctionFailuresRepository;
private static ITrackLogEventsService _trackLogEventsService;
[FunctionName("CreateTempVMTempTablesFunction")]
public static async Task Run([QueueTrigger("%environment-plan%" + AzureFunctionConstants.CreateTempTableQueue, Connection = "AzureWebJobsStorage")]string myQueueItem, ILogger log)
{
string ErrorMessage = string.Empty;
var start = DateTime.Now;
FunctionStatusEnum IsSuccess = FunctionStatusEnum.Success;
log.LogInformation($"C# Queue trigger function processed: {myQueueItem} - {start}");
Guid tempid = new Guid(myQueueItem);
TempVM tempVM = new TempVM();
try
{
container = BusinessLogic.Helpers.SimpleInjectorWebApiInitializer.InitializeSingleton();;
_processingFunctionService = container.GetInstance<IProcessingFunctionService>();
_azureFunctionFailuresRepository = container.GetInstance<IAzureFunctionFailuresRepository>();
_trackLogEventsService = container.GetInstance<ITrackLogEventsService>();
tempVM = await _processingFunctionService.GetById(tempid);
if (tempVM != null)
{
FunctionStatusEnum IsAlreadyPerformed = await _azureFunctionFailuresRepository.GetAzureFunctionFailureStatus(AzureFunctionConstants.CreateTempVMTempTablesFunction, tempVM.Id);
if (IsAlreadyPerformed != FunctionStatusEnum.Success)
{
ResponseData response = await _processingFunctionService.CreateTempVMTempTables(tempid);
}
else
{
ErrorMessage = AzureFunctionConstants.FunctionAlreadyProcessed;
}
}
else
{
ErrorMessage = AzureFunctionConstants.TempVMNotFound;
}
}
catch (Exception ex)
{
IsSuccess = FunctionStatusEnum.Failed;
ErrorMessage = ex.ToString();
}
finally
{
AzureFunctionFailures azureFunctionFailures = new AzureFunctionFailures()
{
Id = Guid.NewGuid(),
FunctionName = AzureFunctionConstants.CreateTempVMTempTablesFunction,
QueueItem = myQueueItem,
ErrorMessage = ErrorMessage,
StartTime = start,
EndTime = DateTime.Now,
FailureTypeId = tempid,
FunctionStatus = IsSuccess,
ProcessTime = (DateTime.Now - start).TotalMilliseconds,
};
await _azureFunctionFailuresRepository.Add(azureFunctionFailures);
}
log.LogInformation($"End Time : {DateTime.Now} - QueueItem {myQueueItem}");
log.LogInformation($"Total Time : {DateTime.Now - start} - QueueItem {myQueueItem}");
}
}
I have check the code where added entry in queue, but only one entry is added for one queueitem.
This issue happened when added multiple entry in same queue (i.e. load testing where I have added 24 request only) for different queue item, when single queue is run then this is not happened, our functions are in App Service Plan with autoscaling
As mentioned in comments. Just set value of WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT as 1 in application settings of your function.
Related
I have couple of tables in the DB, one to stores the list of Holidays(HolidayWeeks) in the year and other one to store the setting(Settings) like when an application should be running. Both the tables are like below in the Azure DB
I started to build the Timer Triggered AzureFunction to check everyday at 5 AM if today doesnot fall under the Holiday list as well Today (Day of the week ) is equal to the Order day in the Settings table like below
public static class StartorStopSMAppService
{
[FunctionName("StartorStopSMAppService")]
public static void Run([TimerTrigger("0 0 5 * * *")]TimerInfo myTimer, ILogger log)
{
int holidaychk = 0;
var str = Environment.GetEnvironmentVariable("SurplusMouseDBConnection");
using (SqlConnection conn = new SqlConnection(str))
{
try
{
conn.Open();
var settingscheck = "SELECT TOP 1 [EndTime],[StartTime],[OrderDay]"+
"FROM[dbo].[Settings]"+
"where SUBSTRING(DATENAME(weekday, getdate() AT TIME ZONE 'UTC' AT TIME ZONE 'Eastern Standard Time'), 0, 4) = OrderDay";
var holidaycheck = "SELECT Count(*) FROM[dbo].[HolidayWeeks] where FORMAT(getdate(), 'yyyy-MM-dd') = [HolidateDate]";
SqlCommand cmd1 = new SqlCommand(holidaycheck, conn);
holidaychk = (Int32)cmd1.ExecuteScalar();
if (holidaychk != 0)
{
SqlCommand command = new SqlCommand(settingscheck, conn);
using (SqlDataReader reader = command.ExecuteReader())
{
reader.Read();
if (reader.HasRows)
{
var starttime = reader["StartTime"].ToString();
var endtime = reader["EndTime"].ToString();
// Not sure how to call the start API at this starttime time
}}}}
catch (Exception ex)
{
log.LogInformation(ex.Message);
}
finally
{
if (conn.State != ConnectionState.Closed)
{
conn.Close();
} }}
When the both the conditions are satisfied I need to start the web service on the starttime give on the Settings table which I am storing in the var starttime = reader["StartTime"].ToString(); I understand that I can POST call the API https://learn.microsoft.com/en-us/rest/api/appservice/web-apps/start to start Web App in the App Service. But how can start the application based on starttime variable. I am kind of stuck any help is greatly appreciated
Once your code determines that it is not a holiday and you would like to start the web app at that particular time, you can post a scheduled message to Azure Queue, see this. If you write a Azure Function Queue trigger, as and when it gets invoked, it can call the start REST API for your WebApp.
Its possible with Durable functions as well. You can implement this as below.
Timer trigger function
public static class TimerFunction
{
[FunctionName("ScheduledStart")]
public static async Task RunScheduled([TimerTrigger("0 * * * * *")] TimerInfo timerInfo,[DurableClient] IDurableClient starter,ILogger log)
{
string functionName = "OrchestractionFunction";
string time = "26/10/2021 7:00AM";
var startTime = DateTime.Parse(time).ToUniversalTime();
string instanceId = await starter.StartNewAsync(functionName, Guid.NewGuid().ToString(), startTime);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
}
}
Create a Durable function with delay
`
[FunctionName("OrchestractionFunction")]
public static async Task RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context)
{
var startTime = context.GetInput<DateTime>();
Console.WriteLine(startTime);
await context.CreateTimer(startTime, CancellationToken.None);
await context.CallActivityAsync("ActivityFunction", "Jhon");
}
`
Activity function invoking a Http request
[FunctionName("ActivityFunction")]
public static string SayHello([ActivityTrigger] string name, ILogger log){
log.LogInformation(name);
//HttpRequest here
}
you can extend the orchestration function to call multiple different APIs
The other two answers are good, but using your current code, all you need to do is check if current date is greater than start time, then, perform the request. Here's a sample using Flurl to simplify the request:
public static class StartorStopSMAppService {
[FunctionName("StartorStopSMAppService")]
public static void Run([TimerTrigger("0 0 5 * * *")] TimerInfo myTimer, ILogger log) {
int holidaychk = 0;
var str = Environment.GetEnvironmentVariable("SurplusMouseDBConnection");
using(SqlConnection conn = new SqlConnection(str)) {
try {
conn.Open();
var settingscheck = "SELECT TOP 1 [EndTime],[StartTime],[OrderDay]" +
"FROM[dbo].[Settings]" +
"where SUBSTRING(DATENAME(weekday, getdate() AT TIME ZONE 'UTC' AT TIME ZONE 'Eastern Standard Time'), 0, 4) = OrderDay";
var holidaycheck = "SELECT Count(*) FROM[dbo].[HolidayWeeks] where FORMAT(getdate(), 'yyyy-MM-dd') = [HolidateDate]";
SqlCommand cmd1 = new SqlCommand(holidaycheck, conn);
holidaychk = (Int32) cmd1.ExecuteScalar();
if (holidaychk != 0) {
SqlCommand command = new SqlCommand(settingscheck, conn);
using(SqlDataReader reader = command.ExecuteReader()) {
reader.Read();
if (reader.HasRows) {
var starttime = reader["StartTime"].ToString();
var endtime = reader["EndTime"].ToString();
string subscriptionId = ""; // enter this value
string resourceGroupName = ""; // enter this value
string webAppName = ""; // enter this value
string url = $"https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Web/sites/{webAppName}/start?api-version=2021-02-01";
DateTime dStartTime;
DateTime.TryParse(starttime, out dStartTime);
if (DateTime.UtcNow >= dStartTime){
await url.PostAsync(); // requires Flurl.Http package
}
}
}
}
} catch (Exception ex) {
log.LogInformation(ex.Message);
} finally {
if (conn.State != ConnectionState.Closed) {
conn.Close();
}
}
}
I am trying the fan-out, fan-in pattern. Here's my code
[FunctionName("af_cancellation")]
public static async Task<string> RunOrchestrator(
[OrchestrationTrigger] DurableOrchestrationContext context, ILogger log)
{
var taskList = new List<Task<string>>();
var tokenSource = new CancellationTokenSource();
taskList.Add(context.CallActivityAsync<string>("af_cancellation_Hello", new { ct = tokenSource.Token, city = "Tokyo" }));
taskList.Add(context.CallActivityAsync<string>("af_cancellation_Hello", new { ct = tokenSource.Token, city = "Seattle" }));
taskList.Add(context.CallActivityAsync<string>("af_cancellation_Hello", new { ct = tokenSource.Token, city = "London" }));
try
{
await Task.WhenAll(taskList);
}
catch (FunctionException)
{
log.LogError("trigger function failed");
tokenSource.Cancel();
}
string output = "";
foreach (var t in taskList)
{
output += t.Result;
}
return output;
}
I would like to cancel all the tasks in taskList if any of them throw an exception. What i am noticing is that await Task.WhenAll finishes all the tasks before moving forward.
Here's the sample trigger function
[FunctionName("af_cancellation_Hello")]
public static string SayHello([ActivityTrigger] DurableActivityContext context, ILogger log)
{
var data = context.GetInput<dynamic>();
var name = (string)data.city;
// unable to de-serialize cancellation token here but we'll ignore that.
var ct = JsonConvert.DeserializeObject<CancellationToken>(data.ct);
if (name != "London")
{
System.Threading.Thread.Sleep(1000 * 30);
}
else
{
System.Threading.Thread.Sleep(1000 * 10);
throw new FunctionException("don't like london");
}
log.LogInformation($"Saying hello to {name}.");
return $"Hello {name}!";
}
How can i achieve this?
According to this I think it's not possible. If you need to undo the job from the other activities that succeeded, you must take a look on the Saga Pattern and launch compensating activities.
more info:
https://microservices.io/patterns/data/saga.html
https://www.enterpriseintegrationpatterns.com/patterns/conversation/CompensatingAction.html
I have a problem with the Azure Table Storage. What I'm trying to achieve is saving the ChangeToken of the SharePoint list in order to use the webhooks properly.
Here is the code:
public class TablesHelper
{
private static readonly string TokenTableName = "TokenTable";
public static async Task<ListChangeToken> GetChangeTokenForListAsync(string listId)
{
var retrieveOperation = TableOperation.Retrieve<ListChangeToken>("Lists", listId, new List<string>() { "ChangeToken" });
var tableReference = await GetTableReferenceAsync(TokenTableName);
var tableResult = await tableReference.ExecuteAsync(retrieveOperation);
if(tableResult.Result != null)
{
return tableResult.Result as ListChangeToken;
}
return null;
}
public static async Task SaveChangeTokenForListAsync(ListChangeToken changeToken)
{
var insertOperation = TableOperation.Insert(changeToken);
var tableReference = await GetTableReferenceAsync(TokenTableName);
var result = await tableReference.ExecuteAsync(insertOperation);
}
private static async Task<CloudTable> GetTableReferenceAsync(string tableName)
{
var storageAccount = CloudStorageAccount.Parse(ConfigurationHelper.CloudStorage);
var tableClient = storageAccount.CreateCloudTableClient();
var reference = tableClient.GetTableReference(tableName);
await reference.CreateIfNotExistsAsync();
return reference;
}
}
The ListChangeToken class:
public class ListChangeToken : TableEntity
{
public ListChangeToken(string listId, string changeToken)
{
this.PartitionKey = "Lists";
this.RowKey = listId;
this.ChangeToken = changeToken;
}
public ListChangeToken() { }
public string ChangeToken { get; set;}
}
As per request, the function calling TablesHelper:
[FunctionName("EventHandler")]
public static async Task Run([QueueTrigger("events", Connection = "CloudStorage")]string myQueueItem, TraceWriter log)
{
var notificationGroup = Newtonsoft.Json.JsonConvert.DeserializeObject<NotificationGroup>(myQueueItem);
var contextHelper = new ContextHelper();
foreach (var notification in notificationGroup.Value)
{
UriBuilder uriBuilder = new UriBuilder();
uriBuilder.Scheme = "https";
uriBuilder.Host = ConfigurationHelper.TenantDomain;
uriBuilder.Path = notification.SiteUrl;
using (var ctx = contextHelper.GetAppOnlyContext(uriBuilder.ToString()))
{
//Read change token
var currentChangeToken = await TablesHelper.GetChangeTokenForListAsync(notification.Resource);
if(currentChangeToken == null)
{
log.Error($"No change token found for list {notification.Resource}. This is a NO GO. Please use the '/api/Setup' function.");
}
var listId = Guid.Parse(notification.Resource);
var changes = await CSOMHelper.GetListItemChangesAsync(ctx, listId, currentChangeToken.ChangeToken);
if(changes.Count > 0)
{
var lastChange = changes[changes.Count - 1];
//Save the last change token
var changeTokenValue = lastChange.ChangeToken.StringValue;
await TablesHelper.SaveChangeTokenForListAsync(new ListChangeToken(
notification.Resource,
changeTokenValue
));
await HandleChanges(ctx, changes);
}
}
}
log.Info($"C# Queue trigger function processed: {myQueueItem}");
}
The problem is that always, when using the "GetChangeTokenForListAsync" the Entity is received properly, but the .ChangeToken property is always null. It is also not visible when browsing with the Azure Storage Explorer. What am I doing wrong here?
The issue is related to the Azure Storage Emulator (V. 5.7.0.0). The same code works perfectly when working with the "live" Azure.
I am using Azure Function to send out message to multiple Event Hub outputs (from an input EH). My code is the following:
[FunctionName("Gateway")]
public static void Run([EventHubTrigger("%INPUT_HUB_NAME%", Connection = "iothubconnection", ConsumerGroup = "functiontest")]EventData[] eventHubMessage,
[EventHub("%OUT_HUB_NAME%", Connection = "eventhuboutput")]out EventData outputEventHubMessageHotPath,
[EventHub("%OUT_HUB_NAME%", Connection = "eventhuboutput2")]out EventData outputEventHubMessageColdPath,
TraceWriter log)
{
log.Info("**-- Start Azure Func -- **");
foreach (var ehMsg in eventHubMessage)
{
//section to build up the raw section
var rawMessageSection = GetPayload(ehMsg.GetBytes());
var deviceId = GetDeviceId(ehMsg);
log.Info($"Extracted deviceId: {deviceId}");
if (rawMessageSection.aggregates != null)
{
var message = CreateEHMessages("aggregates", rawMessageSection, deviceId, log);
outputEventHubMessageHotPath = message;
outputEventHubMessageColdPath = message;
}
if (rawMessageSection.events != null)
{
outputEventHubMessageColdPath = CreateEHMessages("events", rawMessageSection, deviceId, log);
}
if (rawMessageSection.ipis != null)
{
outputEventHubMessageColdPath = CreateEHMessages("ipis", rawMessageSection, deviceId, log);
}
if (rawMessageSection.errors != null)
{
outputEventHubMessageColdPath = CreateEHMessages("errors", rawMessageSection, deviceId, log);
}
if (rawMessageSection.batteries != null)
{
outputEventHubMessageColdPath = CreateEHMessages("batteries", rawMessageSection, deviceId, log);
}
//await Task.WhenAll(tasks);
}
outputEventHubMessageHotPath = outputEventHubMessageColdPath = null;
}
where:
public static EventData CreateEHMessages(string messageType, dynamic messageBatch, string deviceId, TraceWriter log)
{
var timezone = messageBatch.timezone;
var deviceInstanceId = messageBatch.deviceInstanceId;
int i = 0;
List<dynamic> result = new List<dynamic>();
foreach (var msg in messageBatch[messageType])
{
msg.deviceId = deviceId;
msg.timezone = timezone;
msg.deviceInstanceId = deviceInstanceId;
msg.type = messageType;
result.Add(msg);
i++;
}
var eventData = new EventData(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(result)));
eventData.PartitionKey = deviceInstanceId;
return eventData;
}
The problem is that this Function does NOT seem to publish to EventHub. I have tried to use different syntax of binding but i cannot get it to work.
I suspect is something to do with output bindings but I again tried many options.
Any idea?
You set out parameter multiple times, so all but the last assignments will be lost. But your last assignment is setting them to null, which essentially means you return no messages from your Function.
Have a look at ICollector instead.
Define your output parameters as Collectors:
[EventHub("%OUT_HUB_NAME%", Connection = "eventhuboutput")]
ICollector<EventData> outputEventHubMessageHotPath,
Then add every message to the Collectors, e.g.:
if (rawMessageSection.events != null)
{
outputEventHubMessageHotPath.Add(
CreateEHMessages("events", rawMessageSection, deviceId, log));
}
Below is my synchronous implementation of the IMessageSessionHandler interface. One problem I have is that when there are no messages in the queue the InitializeReceiverSync method will repeatedly run until the web job ungracefully times out in Azure, resulting in a failure. On the other hand if there are messages in the queue to be read, once they are read, the session closes and the program exits.
What is the best way to exit the method if there are no messages in the queue?
public void ReceiveMessagesFromAzure()
{
int i = 0;
while (i < 4)
{
var QueueName = ConfigurationManager.AppSettings["QueueName"];
var connectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"];
CancellationTokenSource cts = new CancellationTokenSource();
InitializeReceiverSync(connectionString, QueueName, cts.Token);
i++;
}
}
public void InitializeReceiverSync(string connectionString, string queueName, CancellationToken ct)
{
var receiverFactory = MessagingFactory.CreateFromConnectionString(connectionString);
ct.Register(() => receiverFactory.Close());
var client = receiverFactory.CreateQueueClient(queueName, ReceiveMode.PeekLock);
client.RegisterSessionHandler(
typeof(SessionHandler),
new SessionHandlerOptions
{
MessageWaitTimeout = TimeSpan.FromSeconds(5),
MaxConcurrentSessions = 1,
AutoComplete = false
});
}
class SessionHandler : IMessageSessionHandler
{
AzureRepository ar = new AzureRepository([queueName], [connectionString]);
void IMessageSessionHandler.OnMessage(MessageSession session, BrokeredMessage message)
{
ar.ProcessMessage(message);
}
void IMessageSessionHandler.OnCloseSession(MessageSession session)
{
Console.WriteLine("Session closed");
Environment.Exit(0);
}
void IMessageSessionHandler.OnSessionLost(Exception exception)
{
Console.WriteLine(exception.Message);
Console.WriteLine("Press any key to exit");
}
}