I have some code that my Asp.net Core Web API uses to log certain events to a Kafka server running in Confluent Cloud. When I run the API server on my local machine, it can send and receive with Kafka just fine, but when it is running on an Azure App Service, I receive "Local: Message Timed Out" errors. Is there something about Azure App Service networking that I can modify to make the Kafka network traffic flow correctly?
Here is a snippet of the code below:
public class ConfluentKafkaService {
private readonly ClientConfig clientConfig = new ClientConfig
{
BootstrapServers = "...",
ClientId = Dns.GetHostName(),
SecurityProtocol = SecurityProtocol.SaslSsl,
SaslMechanism = SaslMechanism.Plain,
SaslUsername = "...",
SaslPassword = #"..."
};
public async Task SendDeviceEvent(DeviceEvent de) {
var config = new ProducerConfig(clientConfig);
string topicName = $"...";
using var producer = new ProducerBuilder<Null, DeviceEvent>(config)
.Build();
try {
await producer.ProduceAsync(topicName, new Message<Null, DeviceEvent> { Value = de });
}
catch (ProduceException<Null, string> e) {
Console.WriteLine($"Error producing message: {e.Message}");
}
}
}
My connectivity issue was ultimately caused because Azure App Service does not expose its Trusted Certificate store to librdkafka correctly. I downloaded cacert.pem from https://curl.haxx.se/docs/caextract.html and pointed to it by setting SslCaLocation in my ClientConfig like so:
private readonly ClientConfig clientConfig = new ClientConfig
{
BootstrapServers = "",
ClientId = Dns.GetHostName(),
SecurityProtocol = SecurityProtocol.SaslSsl,
SslCaLocation = Path.Combine("assets", "cacert.pem"),
SaslMechanism = SaslMechanism.Plain,
SaslUsername = ""
SaslPassword = ""
}
For further information, see this issue: https://github.com/confluentinc/confluent-kafka-dotnet/issues/1112
Related
Previously I have Azure Web App (.net core) and It successfully track the SQL Server and Service Bus dependency into Application Insights. It is not working some how with Azure Functions.
Environment
dotnet 6
dotnet-isolated mode
log level default set to "Information".
Azure Environment using Consumption plan for Azure Functions.
Application Insights key is configured.
I have Azure API management at front and backend is Azure Function and that call SQL Server and Service Bus.
Api Management Service to Azure function dependency successfully resolved but Azure Function to other component is not working.
I know I am posting my own answer. Also there are chance that in future there may be some good solution or it get integrated the way it is in in-process mode.
By then follow steps.
Add Package
Microsoft.ApplicationInsights.WorkerService
In program.cs in configuring host.
services.AddApplicationInsightsTelemetryWorkerService();
More info at
https://learn.microsoft.com/en-us/azure/azure-monitor/app/worker-service
The only way I've managed to solve this issue so far was by setting up custom Middleware
.ConfigureFunctionsWorkerDefaults(config =>
{
config.UseMiddleware<AiContextMiddleware>();
})
In the IServiceCollection you need to setup simply
.AddApplicationInsightsTelemetryWorkerService()
public class AiContextMiddleware : IFunctionsWorkerMiddleware
{
private readonly TelemetryClient _client;
private readonly string _hostname;
public AiContextMiddleware(TelemetryClient client)
{
_client = client;
_hostname = Environment.GetEnvironmentVariable("AI_CLOUD_ROLE_NAME");
}
public async Task Invoke(FunctionContext context, FunctionExecutionDelegate next)
{
var operationId = ExtractOperationId(context.TraceContext.TraceParent);
// Let's create and start RequestTelemetry.
var requestTelemetry = new RequestTelemetry
{
Name = context.FunctionDefinition.Name,
Id = context.InvocationId,
Properties =
{
{ "ai.cloud.role", _hostname},
{ "AzureFunctions_FunctionName", context.FunctionDefinition.Name },
{ "AzureFunctions_InvocationId", context.InvocationId },
{ "AzureFunctions_OperationId", operationId }
},
Context =
{
Operation =
{
Id = operationId,
ParentId = context.InvocationId,
Name = context.FunctionDefinition.Name
},
GlobalProperties =
{
{ "ai.cloud.role", _hostname},
{ "AzureFunctions_FunctionName", context.FunctionDefinition.Name },
{ "AzureFunctions_InvocationId", context.InvocationId },
{ "AzureFunctions_OperationId", operationId }
}
}
};
var operation = _client.StartOperation(requestTelemetry);
try
{
await next(context);
}
catch (Exception e)
{
requestTelemetry.Success = false;
_client.TrackException(e);
throw;
}
finally
{
_client.StopOperation(operation);
}
}
private static string ExtractOperationId(string traceParent)
=> string.IsNullOrEmpty(traceParent) ? string.Empty : traceParent.Split("-")[1];
}
It's definitely not a perfect solution as you then get two starting logs, but as end result, you get all logs traces + dependencies correlated to an operation.
I've solved this issue in the first place like that, now I'm revisiting whether there are any better ways to solve this.
Let me know too whether you managed to solve this issue on your side.
When I inject IConfiguration in a function, it does not find any keys that only live in my "Azure App Configuration".
I have a functionApp (V3) that accesses App Configuration using the DefaultAzureCredential. I am running this locally in debug hence the need for a default credential. I also have multiple Tenants so I had to set the VisualStudioTenantId and SharedTokenCacheTenantId on DefaultAzureCredentialOptions. My Visual studio user was also given the role "App Configuration Data Reader" to be able to debug.
When connecting to App configuration I get no errors.
Editedto add: I have setup AppConfiguration to authenticate with AzureAD.
See code below:
public override async void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
{
var credOptions = new DefaultAzureCredentialOptions();
var tenantId = Environment.GetEnvironmentVariable("Tenant_Id");
credOptions.VisualStudioTenantId = tenantId;
credOptions.SharedTokenCacheTenantId = tenantId;
var cred = new DefaultAzureCredential(credOptions);
/*Works but requires SharedTokenCacheTenantId*/
var secretClient = new SecretClient(new Uri(vaultURI), cred);
var secret = await secretClient.GetSecretAsync("<secret name>");
/*Works but where are my keys when I try to access them?*/
builder.ConfigurationBuilder.AddAzureAppConfiguration(options =>
{
options.Connect(new Uri(appConfigURI), cred);
}).Build(); //Should I be building this??
}
In my function
public FunctName(IConfiguration configuration)
{
_configuration = configuration;
}
And when I access the property
var prop = _configuration["PropertyName"];
There is an example function app that uses IFunctionsConfigurationBuilder here https://github.com/Azure/AppConfiguration/blob/main/examples/DotNetCore/AzureFunction/FunctionApp/Startup.cs . I would recommend taking a look and seeing if there are any missing pieces.
The title mentions "using DefaultAzureCredential on local". Does that mean that this works as expected if you use a connection string?
Notice the async void ConfigureAppConfiguration. This caused my ConfigureAppConfiguration to not execute synchronously, causing configure to add my App Configuration before it was populated.
I am new to Ignite and Kubernetes. I have a.Net Core 3.1 web application which is hosted Azure Linux App Service.
I followed the instructions (Apache Ignite Instructions Offical Site) and Apache Ignite could run on Azure Kubernetes. I could create a sample table and read-write actions worked successfully. Here is the screenshot of my success tests on PowerShell.
Please see the success test
Now, I try to connect Apache Ignite from my .net core web app but I couldn't make it.
My code is as below. I try to connect with IgniteConfiguration and SpringCfgXml, but both of them getting error.
private void Initialize()
{
var cfg = GetIgniteConfiguration();
_ignite = Ignition.Start(cfg);
InitializeCaches();
}
public IgniteConfiguration GetIgniteConfiguration()
{
var appSettingsJson = AppSettingsJson.GetAppSettings();
var igniteNodes = appSettingsJson["AppSettings:IgniteNodes"];
var nodeList = igniteNodes.Split(",");
var config = new IgniteConfiguration
{
Logger = new IgniteLogger(),
DiscoverySpi = new TcpDiscoverySpi
{
IpFinder = new TcpDiscoveryStaticIpFinder
{
Endpoints = nodeList
},
SocketTimeout = TimeSpan.FromSeconds(5)
},
IncludedEventTypes = EventType.CacheAll,
CacheConfiguration = GetCacheConfiguration()
};
return config;
}
The first error I get:
Apache.Ignite.Core.Common.IgniteException HResult=0x80131500
Message=Java class is not found (did you set IGNITE_HOME environment
variable?):
org/apache/ignite/internal/processors/platform/PlatformIgnition
Source=Apache.Ignite.Core
Also, I have no idea what I am gonna set for IGNITE_HOME, and username and secret to authentication.
Solution :
I finally connect the Ignite on Azure Kubernetes.
Here is my connection method.
public void TestConnection()
{
var cfg = new IgniteClientConfiguration
{
Host = "MyHost",
Port = 10800,
UserName = "user",
Password = "password"
};
using (IIgniteClient client = Ignition.StartClient(cfg))
{
var employeeCache1 = client.GetOrCreateCache<int, Employee>(
new CacheClientConfiguration(EmployeeCacheName, typeof(Employee)));
employeeCache1.Put(1, new Employee("Bilge Wilson", 12500, 1));
}
}
To find to host IP, user name and client secret please check the below images.
Client Id and Secret
IP Addresses
Note: I don't need to set any IGNITE_HOME ana JAVA_HOME variable.
The simplest way is to download Apache Ignite binary distribution (of the same version as one that you use), unzip it to a directory, and point IGNITE_HOME environment variable or IgniteConfiguration.IgniteHome configuration property to unzipped apache-ignite-n.n.n-bin/ directory absolute path.
We support doing that automatically for Windows-hosted apps but not for Linux-based deployments.
I'm trying to develop WebJob using SDK 3.0.x, and testing it locally. I've followed the sample in github without any success.
When running it locally everything is going ok, it also see the ProcessQueueMessage function but it doesn't pick the messages from the queue.
Program.cs
static void Main(string[] args)
{
var builder = new HostBuilder();
//builder.UseEnvironment(EnvironmentName.Development);
builder.ConfigureWebJobs(b =>
{
b.AddAzureStorageCoreServices();
b.AddAzureStorage();
});
builder.ConfigureAppConfiguration((context, config) =>
{
config.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
});
builder.ConfigureLogging((context, b) =>
{
b.AddConsole();
// If the key exists in settings, use it to enable Application Insights.
string instrumentationKey = context.Configuration["APPINSIGHTS_INSTRUMENTATIONKEY"];
if (!string.IsNullOrEmpty(instrumentationKey))
{
b.AddApplicationInsights(o => o.InstrumentationKey = instrumentationKey);
}
});
builder.ConfigureServices((context, services) =>
{
//services.AddSingleton<IJobActivator, MyJobActivator>();
services.AddScoped<Functions, Functions>();
services.AddSingleton<IHostService, HostService>();
})
.UseConsoleLifetime();
var host = builder.Build();
using (host)
{
host.Run();
}
}
Functions.cs
public class Functions
{
private readonly IHostService _hostService;
public Functions(IHostService hostService)
{
_hostService = hostService;
}
// This function will get triggered/executed when a new message is written
// on an Azure Queue called queue.
public void ProcessQueueMessage([QueueTrigger("newrequests")] string dd,
//DateTimeOffset expirationTime,
//DateTimeOffset insertionTime,
//DateTimeOffset nextVisibleTime,
//string queueTrigger,
//string id,
//string popReceipt,
//int dequeueCount,
ILogger logger)
{
var newRequestItem = new RequestQueueItem();
logger.LogTrace($"New queue item received...");
//logger.LogInformation($" QueueRef = {id} - DequeueCount = {dequeueCount} - Message Content [Id = {newRequestItem.Id}, RequestDate = {newRequestItem.RequestDate}, Mobile = {newRequestItem.Mobile}, ProviderCode = {newRequestItem.ProviderCode}, ItemIDClass = {newRequestItem.MappingIDClass}]");
// TODO: Read the DatabaseConnectionString from App.config
logger.LogTrace($" Getting DB ConnectionString...");
var connectionString = ConfigurationManager.ConnectionStrings["DatabaseConnection"].ConnectionString;
// TODO: Initiation of provider service instance
logger.LogTrace($" Init IalbayanmtnclientserviceClient service instance...");
var bayanService = new AlbayanMtnWCFService.IalbayanmtnclientserviceClient();
// TODO: sending request to provider service endpoint and wait for response
logger.LogTrace($" Sending request to Service Endpoint...");
var response= bayanService.requestpaymenttransactionAsync("agentcode", "agentpassword", "accountno", int.Parse(newRequestItem.TransactionType), newRequestItem.MappingIDClass, newRequestItem.Mobile, (int)newRequestItem.Id).Result;
logger.LogTrace($"Done processing queue item");
}
}
Here is the screen shot for the output
Appreciate your help
Screen shot for queue messages 'newrequests'
enter image description here
From your snapshot, your webjob runs well on local. It didn't pick message because you don't add message in the newrequests queue.
The function only be triggered after you add the message. Or I will get the same result just like yours.
About the tutorial , your could refer to the official doc:Get started with the Azure WebJobs SDK. And make sure you set the right storage account. The below is my appsettings.json. Make sure the "Copy to output directory" property of the appSettings.json file is set to either Copy if newer or Copy always. Or it will run into exception:Storage account 'Storage' is not configured.
{
"ConnectionStrings": {
"AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=mystorage;AccountKey=key;..."
}
}
Hope this could help you, if you still have other questions, please let me know.
I created a simple ASP.NET Core Web application using OAuth authentication from Google. I have this running on my local machine fine.
Yet after deploying this as an AppService to Azure the OAuth redirects seem to get messed up.
The app itself can be found here:
https://gcalworkshiftui20180322114905.azurewebsites.net/
Here's an url that actually returns a result and shows that the app is running:
https://gcalworkshiftui20180322114905.azurewebsites.net/Account/Login?ReturnUrl=%2F
Sometimes the app responds fine but once I try to login using Google it keeps loading forever and eventually comes back with the following message:
The specified CGI application encountered an error and the server terminated the process.
Behind the scenes, the authentication callback that seems to be failing with a 502.3 error:
502.3 Bad Gateway “The operation timed out”
The error trace can be found here:
https://gcalworkshiftui20180322114905.azurewebsites.net/errorlog.xml
The documentation from Microsoft hasn't really helped yet.
https://learn.microsoft.com/en-us/azure/app-service/app-service-authentication-overview
Further investigation leads me to believe that this has to do with the following code:
public GCalService(string clientId, string secret)
{
string credPath = System.Environment.GetFolderPath(System.Environment.SpecialFolder.Personal);
credPath = Path.Combine(credPath, ".credentials/calendar-dotnet-quickstart.json");
var credential = GoogleWebAuthorizationBroker.AuthorizeAsync(
new ClientSecrets
{
ClientId = clientId,
ClientSecret = secret
},
new[] {CalendarService.Scope.Calendar},
"user",
CancellationToken.None,
new FileDataStore(credPath, true)).Result;
// Create Google Calendar API service.
_service = new CalendarService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "gcalworkshift"
});
}
As I can imagine Azure not supporting personal folders? Googling about this doesn't tell me much.
I followed Facebook, Google, and external provider authentication in ASP.NET Core and Google external login setup in ASP.NET Core to create a ASP.NET Core Web Application with Google authentication to check this issue.
I also followed .NET console application to access the Google Calendar API and Calendar.ASP.NET.MVC5 to build my sample project. Here is the core code, you could refer to them:
Startup.cs
public class Startup
{
public readonly IDataStore dataStore = new FileDataStore(GoogleWebAuthorizationBroker.Folder); //C:\Users\{username}\AppData\Roaming\Google.Apis.Auth
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}
public IConfiguration Configuration { get; }
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
services.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")));
services.AddIdentity<ApplicationUser, IdentityRole>()
.AddEntityFrameworkStores<ApplicationDbContext>()
.AddDefaultTokenProviders();
services.AddAuthentication().AddGoogle(googleOptions =>
{
googleOptions.ClientId = "{ClientId}";
googleOptions.ClientSecret = "{ClientSecret}";
googleOptions.Scope.Add(CalendarService.Scope.CalendarReadonly); //"https://www.googleapis.com/auth/calendar.readonly"
googleOptions.AccessType = "offline"; //request a refresh_token
googleOptions.Events = new OAuthEvents()
{
OnCreatingTicket = async (context) =>
{
var userEmail = context.Identity.FindFirst(ClaimTypes.Email).Value;
var tokenResponse = new TokenResponse()
{
AccessToken = context.AccessToken,
RefreshToken = context.RefreshToken,
ExpiresInSeconds = (long)context.ExpiresIn.Value.TotalSeconds,
IssuedUtc = DateTime.UtcNow
};
await dataStore.StoreAsync(userEmail, tokenResponse);
}
};
});
services.AddMvc();
}
}
}
CalendarController.cs
[Authorize]
public class CalendarController : Controller
{
private readonly IDataStore dataStore = new FileDataStore(GoogleWebAuthorizationBroker.Folder);
private async Task<UserCredential> GetCredentialForApiAsync()
{
var initializer = new GoogleAuthorizationCodeFlow.Initializer
{
ClientSecrets = new ClientSecrets
{
ClientId = "{ClientId}",
ClientSecret = "{ClientSecret}",
},
Scopes = new[] {
"openid",
"email",
CalendarService.Scope.CalendarReadonly
}
};
var flow = new GoogleAuthorizationCodeFlow(initializer);
string userEmail = ((ClaimsIdentity)HttpContext.User.Identity).FindFirst(ClaimTypes.Name).Value;
var token = await dataStore.GetAsync<TokenResponse>(userEmail);
return new UserCredential(flow, userEmail, token);
}
// GET: /Calendar/ListCalendars
public async Task<ActionResult> ListCalendars()
{
const int MaxEventsPerCalendar = 20;
const int MaxEventsOverall = 50;
var credential = await GetCredentialForApiAsync();
var initializer = new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "ASP.NET Core Google Calendar Sample",
};
var service = new CalendarService(initializer);
// Fetch the list of calendars.
var calendars = await service.CalendarList.List().ExecuteAsync();
return Json(calendars.Items);
}
}
Before deploying to Azure web app, I changed the folder parameter for constructing the FileDataStore to D:\home, but got the following error:
UnauthorizedAccessException: Access to the path 'D:\home\Google.Apis.Auth.OAuth2.Responses.TokenResponse-{user-identifier}' is denied.
Then, I tried to set the parameter folder to D:\home\site and redeploy my web application and found it could work as expected and the logged user crendentials would be saved under the D:\home\site of your azure web app server.
Azure Web Apps run in a secure environment called the sandbox which has some limitations, details you could follow Azure Web App sandbox.
Additionally, you mentioned about the App Service Authentication which provides build-in authentication without adding any code in your code. Since you have wrote the code in your web application for authentication, you do not need to set up the App Service Authentication.
For using App Service Authentication, you could follow here for configuration, then your NetCore backend can obtain additional user details (access_token,refresh_token,etc.) through an HTTP GET on the /.auth/me endpoint, details you could follow this similar issue. After retrieved the token response for the logged user, you could manually construct the UserCredential, then build the CalendarService.