I’ve been having an issue with a long running Azure Function App. It’s basically reads in a pipe delimeted file, creates an Employee object from the values, get the corresponding employee database record, and do a database insert or update.
Since it’s taking so long and spiking the CPU, some suggested that I read in each line of the file and send it to a Service Bus. Then have another function inside of my Function App to read from the queue and do my record comparison.
I’ve never used Service Bus before but went through and set one up in Azure. Now I’m trying to use the ServiceBus output binding in my Function App to send the message but I’m not getting it to work. I’ve been following along with some different articles I found including the following ones from Microsoft.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus-output?tabs=in-process%2Cextensionv5&pivots=programming-language-csharp#example
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus?tabs=in-process%2Cextensionv5%2Cextensionv3&pivots=programming-language-csharp
I have the ServiceBusConnection added to my local.settings.json file. I pulled the connection string from the Azure Service Bus "Shared Access policies" section. I double checked the settings and made sure that the property name is correct. However, since the connection string is in the return section I don't know how to debug and confirm it's actually pulling the connection string and that it is correct.
This is what I’ve come up with so far but I’m not seeing any messages in my queue.
[FunctionName("ProcessEmployeeInput")]
[return: ServiceBus("myQueueName", Connection = "ServiceBusConnection")]
public string Run(
[BlobTrigger("%InputContainer%/%InputFolder%/{name}.txt", Connection = "StorageConnection")] Stream fileBlob, string name, ILogger log)
{
log.LogInformation($"Azure Function START.");
log.LogInformation($"Processing file {name}.");
string jsonEmployee = string.Empty;
try
{
using (var srFile = new StreamReader(fileBlob))
{
srFile.ReadLine(); //first line is a header. Just call ReadLine to skip past it.
while (!srFile.EndOfStream)
{
var record = srFile.ReadLine();
//Create an Employee object and map the properties.
Employee employee = MapEmployeeProperties(record, log);
//Serialize into text before sending to service bus
jsonEmployee = JsonConvert.SerializeObject(employee);
}
}
}
catch (Exception ex)
{
log.LogError("Error processing file. {0} | {1} | {2}", ex.Message, ex.StackTrace, ex.InnerException);
}
return jsonEmployee;
}
Error Message
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
I didn't see this at first but from this message it sounds like it can't find the Service Bus endpoint. Since the connection is in the output binding, I can confirm it can find it and that it is correct.
Glad that #Caverman for solved the issue. Thank you #Jesse Squire for your suggestions that helped to fix the issue. Posting this on behalf of your discussion so that it will be beneficial for other community members.
Workaround:-
To resolve the above issue make sure that we have following in our host.json file and the nuget package
Azure.Messaging.ServiceBus is installed in the environment.
host.json file example:-
{
"version": "2.0",
"extensions": {
"serviceBus": {
"clientRetryOptions":{
"mode": "exponential",
"tryTimeout": "00:01:00",
"delay": "00:00:00.80",
"maxDelay": "00:01:00",
"maxRetries": 3
},
"prefetchCount": 0,
"transportType": "amqpWebSockets",
"webProxy": "https://proxyserver:8080",
"autoCompleteMessages": true,
"maxAutoLockRenewalDuration": "00:05:00",
"maxConcurrentCalls": 16,
"maxConcurrentSessions": 8,
"maxMessageBatchSize": 1000,
"sessionIdleTimeout": "00:01:00",
"enableCrossEntityTransactions": false
}
}
}
For more information please refer this MICROSOFT DOCUMENTATION FAQ .
Related
I have an Azure Functions in C#. The trigger is a message from a ServiceBus.
When it receives the message, it starts a process to convert same data. The process could be quite long, around 50/60 seconds and the process calls few APIs.
I get this error:
[2023-02-14T00:42:01.683Z] Executed 'getDataSB' (Failed,
Id=ec780726-d740-4445-b964-d8493233874a, Duration=22985ms)
[2023-02-14T00:42:01.685Z] System.Private.CoreLib: Exception while
executing function: getDataSB. System.Net.Http: Response
status code does not indicate success: 302 (Found).
[2023-02-14T00:42:01.747Z] Message processing error
(Action=ProcessMessageCallback, EntityPath=vs,
Endpoint=wb.servicebus.windows.net)
[2023-02-14T00:42:01.748Z]
System.Private.CoreLib: Exception while executing function:
getDataSB. System.Net.Http: Response status code does
not indicate success: 302 (Found).
I don't understand if the error is coming from the APIs or from the Azure Functions. I continue to receive the message again and again. I added a lot of logs but they are not displayed in the local console.
public class GetDataSB
{
private readonly ILogger<GetDataSB> _logger;
public GetReversoVerbSynonymSB(ILogger<GetDataSB> log)
{
_logger = log;
}
[FunctionName("getDataSB")]
public async Task Run(
[ServiceBusTrigger("vs", Connection = "SBConnectionString")]
string myQueueItem)
{
ServiceBusRequest request =
JsonSerializer.Deserialize<ServiceBusRequest>(myQueueItem);
string processName = Environment.GetEnvironmentVariable("ProcessName");
await ProcessService.ProcessData(request, processName, _logger);
}
}
ProcessService is a static class.
public static class ProcessService {
public static async Task<IActionResult> ProcessData(
ServiceBusRequest request, string SystemUser, ILogger log) {
log.LogInformation("Reading data...");
}
}
Is there a way to change the time out of the execution?
I thought to limit the number of concurrency adding in the host.json this setting:
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 1,
"maxAutoRenewDuration": "00:05:00"
},
"sessionHandlerOptions": {
"autoComplete": true,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 1
}
}
}
}
It seems like your Azure Function is failing to process the message received from the ServiceBus and as a result, the same message is being delivered to the function again and again. There could be several reasons for this, such as issues with the APIs that you are calling or issues with the configuration of your Azure Function.
Steps to debug this issue:
Verify the response of the APIs that you are calling from your function.
The 302 response that you are seeing indicates a redirection response. This could be because the API endpoint that you are calling has been moved or is temporarily unavailable. Make sure that you are using the correct API endpoint and that it is available.
Check the logs of your Azure Function. You mentioned that you added a lot of logs, but they are not displayed in the local console. Make sure that you are using the correct logging configuration and that you are looking at the correct log stream. You can also try to use other logging frameworks like Serilog to log the details of the process.
Check the configuration of your Azure Function. It is possible that the function is timing out before it can complete the process.
You can try to increase the timeout value of the Azure Function by modifying the functionTimeout property in the host.json file.
Check the ServiceBus configuration. It is possible that the message is not being properly acknowledged or processed by the Azure Function, causing it to be redelivered.
Use Azure Application Insights to monitor your Azure Function.
This helps you in identifying any performance issues, exceptions, or other problems that may be occurring in your function.
Thanks #Skin for the comments.
Attach the processes with Visual Studio debugger.
Selecting the process to debug.
For more information, refer the below MSDocs.
Attach processes with the debugger
Remote debugging
I use CosmosClient from SDK Microsoft.Azure.Cosmos 3.28.0 in ASP.NET Core 3.1 in Azure Durable Function. This client is getting and sending data from/to my cosmos instance (Core (SQL)) and it works fine but I see that it constantly throws exception in following http request for metadata
GET 169.254.169.254/metadata/instance
System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions.
I use following configuration:
private static void RegisterCosmosDbClient(ContainerBuilder builder)
{
builder.Register(c => new SocketsHttpHandler()
{
PooledConnectionLifetime = TimeSpan.FromMinutes(10), // Customize this value based on desired DNS refresh timer
MaxConnectionsPerServer = 20, // Customize the maximum number of allowed connections
}).As<SocketsHttpHandler>().SingleInstance();
builder.Register(
x =>
{
var cosmosDbOptions = x.Resolve<CosmosDbOptions>();
var socketsHttpHandler = x.Resolve<SocketsHttpHandler>();
return new CosmosClient(cosmosDbOptions.ConnectionString, new CosmosClientOptions()
{
ConnectionMode = ConnectionMode.Direct,
PortReuseMode = PortReuseMode.PrivatePortPool,
IdleTcpConnectionTimeout = new TimeSpan(0, 23, 59, 59),
SerializerOptions = new CosmosSerializationOptions()
{
PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase
},
HttpClientFactory = () => new HttpClient(socketsHttpHandler, disposeHandler: false)
});
})
.AsSelf()
.SingleInstance();
}
I also tried approach with passing IHttpClientFactory from this blog but it didn't help.
It looks like there are no new sockets available in your environment therefore you are getting the socket forbidden error. Please review how to manage connection for Azure Cosmos DB clients and you should use a singleton Azure Cosmos DB client for the lifetime of your application to resolve the issue. In case if you still facing the issue leveraging the singleton object then please let me know so we can further review it.
That particular IP and path is for https://learn.microsoft.com/azure/virtual-machines/windows/instance-metadata-service?tabs=windows
The SDK is attempting to detect the Azure information. It could mean for Durable Functions, this information and endpoint is not available.
This does not affect SDK operations and should not block you from performing other actions on the CosmosClient instance.
I am trying out Azure Function Apps.
The first one following the example in a tutorial with Open Weather map, stopped working after I used log.WriteLine(), which correctly threw a compiler error. I changed to log.Info() and it kept complaining about TraceWriter not containing a definition for WriteLine.
After a lengthy troubleshooting session, I created a new function, copying all the content of the broken one, and it worked immediately.
Created a new function, as before, and began making changes to the Run() method, and running this function yields:
"The resource you are looking for has been removed, had its name
changed, or is temporarily unavailable."
Bearing in mind, the function URL is based on the default key Azure generates when the function is created: https://.azurewebsites.net/api/WeatherWhereYouAre?code=my1really2RAndom3defauLT4Key5from6Azure==
Created yet another function, with no changes from the default "Hello Azure" sample, and it yields a 500 error with:
"Exception while executing function: Functions.HttpTriggerCSharp2 ->
One or more errors occurred. -> Exception binding parameter 'req' ->
Input string was not in a correct format."
This is the content of the project.json file:
{
"frameworks": {
"net46": {
"dependencies": {
"Microsoft.IdentityModel.Clients.ActiveDirectory": "3.16.0",
"Microsoft.Azure.KeyVault": "2.3.2",
"Microsoft.AspNet.WebApi.Client": "5.2.3"
}
}
}
}
And the run.csx:
using System.Net;
public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
log.Info("C# HTTP trigger function processed a request.");
// parse query parameter
string name = req.GetQueryNameValuePairs()
.FirstOrDefault(q => string.Compare(q.Key, "name", true) == 0)
.Value;
// Get request body
dynamic data = await req.Content.ReadAsAsync<object>();
// Set name to query string or body data
name = name ?? data?.name;
return name == null
? req.CreateResponse(HttpStatusCode.BadRequest, "Please pass a name on the query string or in the request body")
: req.CreateResponse(HttpStatusCode.OK, "Hello " + name);
}
EDIT
In the above image, note that this is httpTriggerFSharp1, but the exception is HttpTriggerCSharp2 (which is the only one that works!)
Is there a way I can properly troubleshoot these?
For the default HttpTrigger template for C#, you could call it as follows:
Get https://brucefunapp.azurewebsites.net/api/HttpTriggerCSharp3?name=bruce&code=ItDhLMxwDYmTvMTYzVbbALtL5GEcmaL5DlzSaD4FRIuFdh17ZkY71g==
Or
Post https://brucefunapp.azurewebsites.net/api/HttpTriggerCSharp3?code=ItDhLMxwDYmTvMTYzVbbALtL5GEcmaL5DlzSaD4FRIuFdh17ZkY71g==
Content-type: application/json
{"name": "bruce"}
For more details about Azure Functions C# script, you could refer to here.
Is there a way I can properly troubleshoot these?
Per my understanding, you could leverage Precompiled functions and use Visual Studio 2017 Tools for Azure Functions for creating, local debugging, and publishing to Azure.
Github link for reproduction.
I have an ASP.NET Core (RC1) application that works fine locally. The issue I'm having is that my connection string is not being picked up by my Azure app. I've asked similar questions to this, but I've narrowed down the issue on my end in this app. Note, it requires an app on Azure to reproduce it.
Here's the issue I'm seeing.
First, my configuration is setup as such:
public Startup()
{
var builder = new ConfigurationBuilder()
.AddJsonFile("config.json")
.AddEnvironmentVariables();
mConfiguration = builder.Build();
}
And EF7 is setup here:
public void ConfigureServices(IServiceCollection services)
{
services.AddEntityFramework()
.AddSqlServer()
.AddDbContext<FooDbContext>(options =>
{
// I'm assuming it's failing here.
// I'm not sure how to debug it running on Azure.
// All the developer exception page shows is:
// 500 Internal Server Error "An error occurred while starting the application."
options.UseSqlServer(mConfiguration["Data:ConnectionStringTest:ConnectionString"]);
});
services.AddScoped<IFooDataService, FooSqlDataService>();
}
My config.json has:
{
"Data": {
"ConnectionStringTest": {
"ConnectionString": "Data Source=(localdb)\\mssqllocaldb;Initial Catalog=ConnectionStringTest"
}
}
}
And this should be overridden by the connection string I've setup in Azure:
When going to the Kudu SCM and looking at the environment variables on the Azure web app instance, I see the following:
SQLAZURECONNSTR_Data:ConnectionStringTest:ConnectionString = my_connection_string_here
I am assuming this is the class that is being used under the hood when my environment variable is used at runtime: EnvironmentVariablesConfigurationProvider
Ok here's what I found, and this feels awkward.
It seems that you need to use Data:{my_connection_string_key}:ConnectionString everywhere EXCEPT in Azure. This environment variable converter will construct the proper connection string using this format automatically if the connection string is prefixed with SQLAZURECONNSTR_.
This means when you setup your connection string in Azure, you need to omit EVERYTHING except the key to your connection string. Do not insert Data: or :ConnectionString... simply use {connection_string_key} (refer to the above format) instead. If you include the entire format in your Azure key/value pair, the EnvironmentVariablesConfigurationProvider will add another Data: and :ConnectionString around it, resulting in something like Data:Data:{my_connection_string_key}:ConnectionString:ConnectionString.
In ConfigureServices(...), use the format that ASP expects:
... options.UseSqlServer(mConfiguration["Data:ConnectionStringTest:ConnectionString"]);
You can therefore locally use this for your local JSON, for testing in development/fallback:
{
"Data": {
"ConnectionStringTest": {
"ConnectionString": "Data Source=(localdb)\\mssqllocaldb;Initial Catalog=ConnectionStringTest"
}
}
}
Just make sure your Azure connection string has the middle part of that format (ConnectionStringTest using this example).
This will make your environment variable in Azure look like this in raw format:
SQLAZURECONNSTR_ConnectionStringTest = {insert connection string here}
And the EnvironmentVariablesConfigurationProvider will strip off the Azure prefix string, and wrap your key in the hardcoded format: Data:{0}:ConnectionString
Experiment Results
To augment your excellent answer, I did a local experiment to confirm that the SQLAZURECONNSTR_connection_string_key environmental variable becomes this configuration:
mConfiguration["Data:connection_string_key:ConnectionString"]
Local Experiment
A local environmental variable emulates an Azure SQL Database connection string named connection_string_key.
PS> $env:SQLAZURECONNSTR_connection_string_key = "an azure conn string"
The following code dumps all the environmental variables and configuration sections to the page.
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
app.Run(async (context) =>
{
await context.Response.WriteAsync("# Environmental Variables \r\n");
await DumpAllEnvVariables(context, Environment.GetEnvironmentVariables());
await context.Response.WriteAsync("# Configuration Sections \r\n");
await DumpAllConfigItems(context, mConfiguration.GetChildren());
});
}
private async Task DumpAllEnvVariables(HttpContext context, IDictionary envVariables)
{
foreach (var envVar in envVariables.Cast<DictionaryEntry>())
{
await context.Response.WriteAsync($"{envVar.Key}"); // : {envVar.Value}
await context.Response.WriteAsync($"\r\n");
}
}
private async Task DumpAllConfigItems(HttpContext context,
IEnumerable<IConfigurationSection> sections, string prefix = "")
{
foreach (var section in sections)
{
await context.Response.WriteAsync($"{prefix}{section.Key}"); // : {envVar.Value}
await context.Response.WriteAsync($"\r\n");
if(section.GetChildren().Any())
{
await DumpAllConfigItems(context, section.GetChildren(), prefix + " ");
}
}
}
I’ve got a WinRT app that I’m using the Windows Azure Toolkit for Windows 8 with. I’ve got a setup where I’d like clients subscribed to ignore messages posted to a ServiceBus Topic if they’re the originator or if the message is older than when their subscription started.
In the Properties of my BrokeredMessage, I’ve added 2 items to cover these scenarios:
message.Properties["Timestamp"] = DateTime.UtcNow.ToFileTime();
message.Properties["OriginatorId"] = clientId.ToString();
clientId is a Guid.
The subscriber side looks like this:
// ti is a class that contains a Topic, Subscription and a bool as a cancel flag.
string FilterName = "NotMineNewOnly";
// Find or create the topic.
if (await Topic.ExistsAsync(DocumentId.ToString(), TokenProvider))
{
ti.Topic = await Topic.GetAsync(DocumentId.ToString(), TokenProvider);
}
else
{
ti.Topic = await Topic.CreateAsync(DocumentId.ToString(), TokenProvider);
}
// Find or create this client's subscription to the board.
if (await ti.Topic.Subscriptions.ExistsAsync(ClientSettings.Id.ToString()))
{
ti.Subscription = await ti.Topic.Subscriptions.GetAsync(ClientSettings.Id.ToString());
}
else
{
ti.Subscription = await ti.Topic.Subscriptions.AddAsync(ClientSettings.Id.ToString());
}
// Find or create the subscription filter.
if (!await ti.Subscription.Rules.ExistsAsync(FilterName))
{
// Want to ignore messages generated by this client and ignore any that are older than Timestamp.
await ti.Subscription.Rules.AddAsync(FilterName, sqlFilterExpression: string.Format("(OriginatorId != '{0}') AND (Timestamp > {1})", ClientSettings.Id, DateTime.UtcNow.ToFileTime()));
}
ti.CancelFlag = false;
Topics[boardId] = ti;
while (!ti.CancelFlag)
{
BrokeredMessage message = await ti.Subscription.ReceiveAndDeleteAsync(TimeSpan.FromSeconds(30));
if (!ti.CancelFlag && message != null)
{
// Everything gets here! :(
}
I get back everything – so I’m not sure what I’m doing wrong. What’s the easiest way to troubleshoot problems with subscription filters?
When you create a Subscription then by default you get a "MatchAll" filter. In the code above you are just adding your filter so it is applied in addition to the "MatchAll" filter and thus all messages are recieved. Just delete the $Default filter once the Subscription is created and that should resolve the issue.
Best way to troubleshoot is using the Service Bus Explorer from Paolo Salvatori
http://code.msdn.microsoft.com/windowsazure/Service-Bus-Explorer-f2abca5a
He has done a good few blog posts on it e.g. http://windowsazurecat.com/2011/07/exploring-topics-and-queues-by-building-a-service-bus-explorer-toolpart-1/
Windows Azure SDK 1.7 does have built in capability but the Service Bus Explorer Standalone version is still better, see comparison here.
http://soa-thoughts.blogspot.com.au/2012/06/visual-studio-service-bus-explorer.html
HTH your debugging...