Timer based Azure function with Table storage, HTTP request, and Azure Service Bus - azure

I have a process written in a console application right now that fires on a scheduled task to read data from Azure table storage and based on that data, make API calls to a third party vendor we use, deserialize the response data, loop over an array in the results, save the individual iterations of the loop into a different table in Azure table storage, and then publish messages for each iteration of the loop to Azure service bus where those messages are consumed by another client.
In an effort to get more of our tasks into the cloud, I've done some research and it seems that an Azure function would be a good candidate to replace my console application. I spun up a new Azure function project in Visual Studio 2019 as a "timer" function and then dove into some reading where I got lost really fast.
The reading I've done talks about using "bindings" in my Run() method arguments decorated with attributes for connection strings etc but I'm not sure that is the direction I should be heading. It sounds like that would make it easier for authentication to my table storage, but I can't figure out how to use those "hooks" to query my table and then perform inserts. I haven't even gotten to the service bus stuff yet nor looked into making HTTP calls to our third party vendor's api.
I know this is a very broad question and I don't have any code to post because I'm having a tough time even getting out of the starting blocks with this. The MS documentation is all over the map and I can't find anything specific to my needs and I promise I've spent a fair bit of time trying.
Are Azure functions even the right path I should be travelling? If not, what other options are out there?
TIA

You should keep with Azure Functions with the Time Trigger to replace your console app.
The bindings (which can be used for input /output) are helpers to save you some lines of code, for example:
Rather than using the following code to insert data into azure table:
// Retrieve storage account information from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);
// Create a table client for interacting with the table service
CloudTableClient tableClient = storageAccount.CreateCloudTableClient(new TableClientConfiguration());
// Create a table client for interacting with the table service
CloudTable table = tableClient.GetTableReference("MyTable");
//some code to populate an entity
var entity = new { PartitionKey = "Http", RowKey = Guid.NewGuid().ToString(), Text = input.Text };
// Create the InsertOrReplace table operation
TableOperation insertOrMergeOperation = TableOperation.InsertOrMerge(entity);
// Execute the operation.
TableResult result = await table.ExecuteAsync(insertOrMergeOperation);
you would use:
[FunctionName("TableOutput")]
[return: Table("MyTable")]
public static MyPoco TableOutput([HttpTrigger] dynamic input, ILogger log)
{
log.LogInformation($"C# http trigger function processed: {input.Text}");
return new MyPoco { PartitionKey = "Http", RowKey = Guid.NewGuid().ToString(), Text = input.Text };
}
PS: the input trigger in the previous code is a HTTP Trigger, but was only to explain how to use output binding.
you can find more information in here:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-triggers-bindings
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-table
and you should watch: https://learn.microsoft.com/en-us/learn/modules/chain-azure-functions-data-using-bindings/

Related

Azure Insights: tracking custom property through the chain of function executions

I have Azure Function1->Function2->Service flow of calls in my Azure app. There are multiple concurrent calls of Function1 and each could be identified by some unique input Document Id. I wonder how in c# code I can set something in Azure Insight context to that document id in the beginning of Funciton1, so that any [traces] or [exceptions] or [dependencies] logged to Azure Insights in any of the follow up calls contains the document id. I noticed all of them have customDimension nested list of properties, so maybe somehow add one more property to there. Also if Function1 runs multiple times in parallel, I do not want these document id to be mixed up.
Goal is to be able to track this document id in all kinds of logs with min amount of additional c# code, avoid passing the document id from function to other functions and other services, so looking into any type of log (wheatear it's traces or exceptions or other) I'm able to immediately identify document the execution belonged to. Is it possible?
To attach a custom property to all logs whithin an azure function is not that diffucult, one could simply use a telemetry initializer to do that:
public class TelemetryEnrichment : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
if (!(telemetry is ISupportProperties item)) return;
// Demonstrate static property
item.Properties["Environment"] = "Production";
}
}
If it is an http triggered function you can enricht the request telemetry like this:
var requestTelemetry = req.HttpContext.Features.Get<RequestTelemetry>();
requestTelemetry.Properties.Add("aProp", "aValue");
You also want to have to property logged by other functions that are called by the entry function. This is not easy doable: you will need to pass the id to the other function manually, for example by passing it using the url of that function.
However, if you have to Id attached to the logs of the entry function you can easily create a query to correlate the logs. Based on the operation id you can get the whole picture of the communication flow between the functions and services, see the docs:
That way, you do not need to include the Id as a custom property to each and every telemetry item.

Can Azure function GridTrigger be used along with CosmosDB input binding?

I am a newbie to Azure functions. I am having problem with the below code where CosmosDB binding is not working. If I remove CosmosDB, it works find; I am able to get messages thru eventGridEvent. But as soon as I add CosmosDB, this piece of code fails. I had tried different Collectors (IAsyncCollectory, IReadOnlyCollection, IReadOnlyList, IEnumerable, etc.) some of them compiles fine but when deployed to Azure portal it does nothing (msg: function not found).
I need to retrieve data from CosmosDb and save it to a Queue by using the message coming thru eventGridEvent. I can hardcoded but that is not allowed for my project. Any help will be appreciated team!
[FunctionName("TopicGridTrigger")]
public static void Run([EventGridTrigger()] EventGridEvent eventGridEvent,
[Queue("myqueuetest")] out string queueMessage,
[CosmosDB("MyCarStore", "cars", ConnectionStringSetting = "CosmosDBConnectionString")] IAsyncCollector<MyCar> Items,
ILogger log
)
{
//SOME CODE HERE
}

How to save MS bot state data and conversations ?

I am using node js and Microsoft Bot builder sdk to write a BOT. Currently, I am saving everything in data bags (conversationData, userData etc) and using CosmosDb to store the state. I followed this article to configure CosmosDB & made changes as per nodejs.
https://azure.microsoft.com/en-in/blog/bot-conversation-history-with-azure-cosmos-db/
There are few concerns with this approach,
conversationData bag is cleared when we call endConversation() in dialog. Thats expected by sdk design but we would like to persist this data for multiple conversation flows with same user (same conversation id.) Now, json cosmosDb in db gets replaced with new keys on conversationData when user start new intent.
Ex: schedule a meeting with {name} for {day} at {place}.
we save conversationData.name , conversationData.day , and conversationData. place.
same user starts over schedule a meeting with {name2} for {day2} at {place2}.
documentDb entry gets replaced with conversationData.name1 , conversationData.day2 , and conversationData. place2
Ideally, we would like to keep everything.
Is there a better way to save chat history & conversationData, userData
databags in MS BOT ?.
All storage implementations just write getData and saveData, they internally have a key:value store where key is typically userId + conversationId, but you can make it whatever you want as long as you can reliably derive it from the arguments passed to getData and setData.
Looking at a sample in redis https://github.com/suttna/botbuilder-redis-storage - https://github.com/suttna/botbuilder-redis-storage/blob/master/src/storage.ts for an example storage implementation that's pretty easy to follow.
You would use a custom implementation like this
// Create new storage with redis client
var storage = new YourStorage()
// this is just here for the sake of initializing a `bot`
var connector = new builder.ChatConnector()
var bot = new builder.UniversalBot(connector)
// Configure bot to use YourStorage
bot.set('storage', storage)
bot.set('persistConversationData', true);
storage is just an object that implements
public saveData(context: IBotStorageContext, data: IBotStorageData, callback?: (err: Error) => void)
public getData(context: IBotStorageContext, callback: (err: Error, data: IBotStorageData) => void)
I totally just copied those signatures from the linked redis module, but they are the same in the BotBuilder source for the default storage - https://github.com/Microsoft/BotBuilder/blob/5cf71c742f27d89e9dbc4076f850122bd6edac11/Node/calling/src/storage/BotStorage.ts
The samples are in typescript. If you are unfamiliar, ignore the bit right after : which indicates the type of a thing.

Azure WebJobs Connection Strings configuration ( AzureWebJobsDashboard?? )

I'm trying to work with Azure Webjobs, I understand the way its works but I don't understand why I need to use two connection strings, one is for the queue for holding the messages but
why there is another one called "AzureWebJobsDashboard" ?
What its purpose?
And where I get this connection string from ?
At the moment I have one Web App and one Webjob at the same solution, I'm experiment only locally ( without publishing anything ), the one thing I got up in the cloud is the Storage account that holds the queue.
I even try to put the same connection string in both places ( AzureWebJobsDashboard,AzureWebJobsStorage) but its throw exception :
"Cannot bind parameter 'log' when using this trigger."
Thank you.
There are two connection strings because the WebJobs SDK writes some logs in the storage account. It gives you the possibility of having one storage account just for data (AzureWebJobsStorage) and the another one for logs (AzureWebJobsDashboard). They can be the same. Also, you need two of them because you can have multiple job hosts using different data accounts but sending logs to the same dashboard.
The error you are getting is not related to the connection strings but to one of the functions in your code. One of them has a log parameter that is not of the right type. Can you share the code?
Okay, anyone coming here looking for an actual answer of "where do I get the ConnectionString from"... here you go.
On the new Azure portal, you should have a Storage Account resource; mine starts with "portalvhds" followed by a bunch of alphanumerics. Click that so see a resource Dashboard on the right, followed immediately by a Settings window. Look for the Keys submenu under General -- click that. The whole connection string is there (actually there are two, Primary and Secondary; I don't currently understand the difference, but let's go with Primary, shall we?).
Copy and paste that in your App.config file on the connectionString attribute of the AzureWebJobsDashboard and AzureWebJobsStorage items. This presumes for your environment you only have one Storage Account, and so you want that same storage to be used for data and logs.
I tried this, and at least the WebJob ran without throwing an error.
#RayHAz - Expanding upon your above answer (thanks)...
I tried this https://learn.microsoft.com/en-us/azure/app-service/webjobs-sdk-get-started
but in .Net Core 2.1, was getting exceptions about how it couldn't find the connection string.
Long story short, I ended up with the following, which worked for me:
appsettings.json, in a .Net Core 2.1 Console app:
{
"ConnectionStrings": {
"AzureWebJobsStorage": "---your Azure storage connection string here---",
"AzureWebJobsDashboard":"---the same connectionstring---"
}
}
... and my Program.cs file...
using System;
using System.IO;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
namespace YourWebJobConsoleAppProjectNamespaceHere
{
public class Program
{
public static IConfiguration Configuration;
static void Main(string[] args)
{
var builder = new ConfigurationBuilder()
.SetBasePath(Path.Combine(AppContext.BaseDirectory))
.AddJsonFile("appsettings.json", true);
Configuration = builder.Build();
var azureWebJobsStorageConnectionString = Configuration.GetConnectionString("AzureWebJobsStorage");
var azureWebJobsDashboardConnectionString = Configuration.GetConnectionString("AzureWebJobsDashboard");
var config = new JobHostConfiguration
{
DashboardConnectionString = azureWebJobsDashboardConnectionString,
StorageConnectionString = azureWebJobsStorageConnectionString
};
var loggerFactory = new LoggerFactory();
config.LoggerFactory = loggerFactory.AddConsole();
var host = new JobHost(config);
host.RunAndBlock();
}
}
}

How to get x-ms-request-id from Azure table storage api call

I getting slow behavior for my azure tablestorage api calls on a windows azure app.I need to get the request id (x-ms-request-id in the response header) for a particular call. Is there a way I can get it using the storageclient api? Does the storage client api even expose this id? If not, is there any other way to get this id?
I am using the api in the following way:
public UserDataModel GetUserData(String UserId)
{
UserDataModel osudm = null;
try
{
var result = (from c in GetServiceContext().OrgUserIdTable
where (c.RowKey == UserId)
select c).FirstOrDefault();
UserDataSource osuds = new UserDataSource(this.account);
osudm = osuds.GetUserData(result.PartitionKey, result.UserName);
}
catch (Exception e)
{
}
return osudm;
}
What you're asking here is more related to WCF Data Services than it is to Windows Azure (the storage client client API uses this). Here is some example code how you can access the response headers:
var tableContext = new MyTableServiceContext(...);
DataServiceQuery<Order> query = tableContext.Orders.Where(o => o.RowKey == "1382898382") as DataServiceQuery<Order>;
IEnumerable<Order> result = query.Execute();
QueryOperationResponse response = result as QueryOperationResponse;
string requestId;
response.Headers.TryGetValue("x-ms-request-id", out requestId);
So what you'll be doing first is simply create your query and cast it to a DataServiceQuery of TType. Then you can call the Execute method on that query and cast it to a QueryOperationResponse. This class will give you access to all headers, including the x-ms-request-id.
Note that in this case you won't be able to use FirstOrDefault, since this doesn't return an IQueryable and you can't cast it to a DataServiceQuery of TType (unless there's an other way to do it using WCF Data Services).
Note: The reason why the call is so slow might be caused by your query. When you query the OrgUserIdTable table, you only filter based on the RowKey. I don't know how much data or partitions you have in that table, but if you don't use the PartitionKey this might have a significant performance impact. You have to know that, by not including the PartitionKey, you'll force a search on all partitions (possibly over multiple servers) which might be causing the call being so slow.
I suggest you take a look at the following real world guidance to get a better insight on how and why partitioning relates to performance in Windows Azure Storage: Designing a Scalable Partitioning Strategy for Windows Azure Table Storage

Resources