I want to know if it is possible to create an Azure Queue (Service bus or Storage Queue) which can be placed in front of a web application and receives http requests very fist.
updates
Thanks for the comments and answers.
I want to process the request without burding IIS. I need to make it possible to process a request in a queue before it reaches IIS.
if it is possible to create an Azure Queue (Service bus or Storage Queue) which can be placed in front of a web application and receives http requests very fist.
We can save the request message to a queue before handing the request in Azure Web App by adding some code. I wrote a C# version sample code which will record the request message to an Azure Storage queue. Steps below are for your reference.
Step 1. Add a http module to your project. In this module, I registered BeginRequest event of HttpApplication and do the message recording job.
public class RequestToQueueModeule : IHttpModule
{
#region IHttpModule Members
public void Dispose()
{
//clean-up code here.
}
public void Init(HttpApplication context)
{
// Below is an example of how you can handle LogRequest event and provide
// custom logging implementation for it
context.BeginRequest += new EventHandler(OnBeginRequest);
}
#endregion
public void OnBeginRequest(Object source, EventArgs e)
{
HttpApplication context = source as HttpApplication;
AddMessageToQueue(context.Request);
}
public void AddMessageToQueue(HttpRequest request)
{
StringBuilder sb = new StringBuilder();
sb.AppendLine(request.HttpMethod + " " + request.RawUrl + " " + request.ServerVariables["SERVER_PROTOCOL"]);
for (int i = 0; i < request.Headers.Count; i++)
{
sb.AppendLine(request.Headers.Keys[i] + ":" + request.Headers[i]);
}
sb.AppendLine();
if (request.InputStream != null)
{
using (StreamReader sr = new StreamReader(request.InputStream))
{
sb.Append(sr.ReadToEnd());
}
}
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("connection string of your azure storage");
// Create the queue client.
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
// Retrieve a reference to a queue.
CloudQueue queue = queueClient.GetQueueReference("queue name which is used to store the request message");
// Create the queue if it doesn't already exist.
queue.CreateIfNotExists();
// Create a message and add it to the queue.
CloudQueueMessage message = new CloudQueueMessage(sb.ToString());
queue.AddMessage(message);
}
}
Step 2. Register upper module in system.webServer node of web.config. Please modify the namespace name where your module placed.
<system.webServer>
<modules runAllManagedModulesForAllRequests="true">
<add name="RequestToQueueModeule" type="[your namespace name].RequestToQueueModeule" />
</modules>
</system.webServer>
I want to process the request without burding IIS. I need to make it possible to process a request in a queue before it reaches IIS.
If you want to process the request in a queue before it reaches IIS, you need to add a proxy in front of Azure Web App. Azure Application Gateway works as a proxy and it can be put in front of Web App. If you only want to log the main information of HTT request, you could use Azure Application Gateway and turn on the Access Log. For more information, link below is for your reference.
Diagnostic logs of Application Gateway
If you want to save all the request message, I am afraid you need to build a custom proxy and log the request by yourself.
Related
I have a scenario in which I am calling RegisterMessageHandler of SubscriptionClient class of Azure Service Bus library.
Basically I am using trigger based approach while receiving the messages from Service Bus in one of my services in Service Fabric Environment as a stateless service.
So I am not closing the subscriptionClient object immediately, rather I am keeping it open for the lifetime of the Service so that it keeps on receiving the message from azure service bus topics.
And when the service needs to shut down(due to some reasons), I want to handle the cancellation token being passed into the service of Service Fabric.
My question is how can I handle the cancellation token in the RegisterMessageHandler method which gets called whenever a new message is received?
Also I want to handle the closing of the Subscription client "Gracefully", i.e I want that if a message is already being processed, then I want that message to get processed completely and then I want to close the connection.
Below is the code I am using.
Currently We are following the below approach:
1. Locking the process of the message using semaphore lock and releasing the lock in finally block.
2. Calling the cancellationToken.Register method to handle cancellation token whenever cancellation is done. Releasing the lock in the Register Method.
public class AzureServiceBusReceiver
{
private SubscriptionClient subscriptionClient;
private static Semaphore semaphoreLock;
public AzureServiceBusReceiver(ServiceBusReceiverSettings settings)
{
semaphoreLock = new Semaphore(1, 1);
subscriptionClient = new SubscriptionClient(
settings.ConnectionString, settings.TopicName, settings.SubscriptionName, ReceiveMode.PeekLock);
}
public void Receive(
CancellationToken cancellationToken)
{
var options = new MessageHandlerOptions(e =>
{
return Task.CompletedTask;
})
{
AutoComplete = false,
};
subscriptionClient.RegisterMessageHandler(
async (message, token) =>
{
semaphoreLock.WaitOne();
if (subscriptionClient.IsClosedOrClosing)
return;
CancellationToken combinedToken = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, token).Token;
try
{
// message processing logic
}
catch (Exception ex)
{
await subscriptionClient.DeadLetterAsync(message.SystemProperties.LockToken);
}
finally
{
semaphoreLock.Release();
}
}, options);
cancellationToken.Register(() =>
{
semaphoreLock.WaitOne();
if (!subscriptionClient.IsClosedOrClosing)
subscriptionClient.CloseAsync().GetAwaiter().GetResult();
semaphoreLock.Release();
return;
});
}
}
Implement the message client as ICommunicationListener, so when the service is closed, you can block the call until message processing is complete.
Don't use a static Semaphore, so you can safely reuse the code within your projects.
Here is an example of how you can do this.
And here's the Nuget package created by that code.
And feel free to contribute!
I have an application composed of two ASP.NET Core apps, app A and app B.
App A makes HTTP calls to App B, and Application Insights automatically correlates this and shows them as a single request. Great!
However, I'm now moving to a more event-based system design, where app A publishes an event to an Azure Event Grid, and app B is set up with a webhook to listen to that event.
Having made that change, the telemetry correlation is broken and it no longer shows up as a single operation.
I have read this documentation: https://learn.microsoft.com/en-us/azure/azure-monitor/app/correlation which explains the theory around correlation headers - but how can I apply this to the Event Grid and get it to forward the correlation headers on to the subscribing endpoints?
The Header pass-trough idea for a custom topic in the AEG has been recently (Oct.10th) unplanned.
However, the headers can be passed via the AEG model to the subscribers in the data object of the event message. This mediation can be done, for example, using the Policies in Azure API Management.
UPDATE:
The following documents can help for manual instrumentation of the webhook endpoint handler (subscriber side) using a custom tracking operations:
Track custom operations with Application Insights .Net SDK
Application Insights API for custom events and metrics
Add two correlation properties to all your events:
public string OperationId { get; set; }
public string OperationParentId { get; set; }
Publisher side: create Dependency and fill up these properties.
private Microsoft.ApplicationInsights.TelemetryClient _telemetryClient;
async Task Publish<TEventData>(TEventData data)
{
var #event = new EventGridEvent
{
Id = Guid.NewGuid().ToString(),
EventTime = DateTime.UtcNow,
EventType = typeof(TEventData).FullName,
Data = data
};
string operationName = "Publish " + #event.EventType;
// StartOperation is a helper method that initializes the telemetry item
// and allows correlation of this operation with its parent and children.
var operation =
_telemetryClient.StartOperation<DependencyTelemetry>(operationName);
operation.Telemetry.Type = "EventGrid";
operation.Telemetry.Data = operationName;
// Ideally, the correlation properties should go in the request headers but
// with the current implementation of EventGrid we have no other way
// as to store them in the event Data.
data.OperationId = operation.Telemetry.Context.Operation.Id,
data.OperationParentId = operation.Telemetry.Id,
try
{
AzureOperationResponse result = await _client
.PublishEventsWithHttpMessagesAsync(_topic, new[] { #event });
result.Response.EnsureSuccessStatusCode();
operation.Telemetry.Success = true;
}
catch (Exception ex)
{
operation.Telemetry.Success = false;
_telemetryClient.TrackException(ex);
throw;
}
finally
{
_telemetryClient.StopOperation(operation);
}
}
Consumer side: create Request and restore correlation.
[FunctionName(nameof(YourEventDataCosumer))]
void YourEventDataCosumer([EventGridTrigger] EventGridEvent #event)
{
var data = (YourEventData)#event.Data;
var operation = _telemetryClient.StartOperation<RequestTelemetry>(
"Handle " + #event.EventType,
data.OperationId,
data.OperationParentId);
try
{
// Do some event processing.
operation.Telemetry.Success = true;
operation.Telemetry.ResponseCode = "200";
}
catch (Exception)
{
operation.Telemetry.Success = false;
operation.Telemetry.ResponseCode = "500";
throw;
}
finally
{
_telemetryClient.StopOperation(operation);
}
}
This works, but not ideal as you need to repeat this code in every consumer. Also, some early log messages (e.g. emitted by constructors of injected services) are still not correlated correctly.
A better approach would be to create a custom EventGridTriggerAttribute (recreate the whole Microsoft.Azure.WebJobs.Extensions.EventGrid extension) and move this code into IAsyncConverter.ConvertAsync().
We use an Azure Service Bus to post all of our requests from our Xamarin mobile app. The Azure Service Bus is bound to an Azure Function which is triggered each time a requests hits the Azure Service Bus.
We have found that we are getting errors from this Azure Function when we send data above a certain size. We can send up to 800 records without a problem but when we send >=850 records we get the following error:
[Error] Exception while executing function:
Functions.ServiceBusQueueTrigger. mscorlib: Exception has been thrown
by the target of an invocation. mscorlib: One or more errors occurred.
A task was canceled.
The service that is being invoked is an ASP.NET Web API RESTful service that saves the data records into a database. This doesn't generate any errors at all.
Here is my Azure Function code.
#r "JWT.dll"
#r "Common.dll"
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using Microsoft.ServiceBus.Messaging;
public static void Run(BrokeredMessage message, TraceWriter log)
{
log.Info($"C# ServiceBus queue trigger function processed message: {message.MessageId}");
if (message != null)
{
Common.Entities.MessageObjectEntity messageObject = message?.GetBody<Common.Entities.MessageObjectEntity>();
string msgType = messageObject?.MessageType;
var msgContent = messageObject?.MessageContent;
log.Info($"Message type: {msgType}");
double timestamp = (DateTime.UtcNow - new DateTime(1970, 1, 1)).TotalSeconds;
string subscriber = "MYSUBSCRIBER";
string privatekey = "MYPRIVATEKEY";
Dictionary<string, object> payload = new Dictionary<string, object>()
{
{"iat", timestamp},
{"subscriber", subscriber}
};
string token = JWT.JsonWebToken.Encode(payload, privatekey, JWT.JwtHashAlgorithm.HS256);
using (var client = new HttpClient())
{
string url = $"http://myexamplewebservices.azurewebsites.net/api/routingtasks?formname={msgType}";
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(subscriber, token);
HttpContent content = new StringContent((string)msgContent, Encoding.UTF8, "application/json");
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = client.PostAsync(new Uri(url), content);
if (response == null)
{
log.Info("Null response returned from request.");
}
else
{
if (response.Result.IsSuccessStatusCode && response.Result.StatusCode == System.Net.HttpStatusCode.OK)
{
log.Info("Successful response returned from request.");
}
else
{
log.Info($"Unsuccessful response returned from request: {response.Result.StatusCode}.");
}
}
}
log.Info("Completing message.");
}
}
This code has been working for several years and works across all our other apps / web sites.
Any ideas why we're getting errors wehen we post large amounts of data to our Azure Service Bus / Azure Function?
It may caused by "new httpclient", there is a limit to how quickly system can open new sockets so if you exhaust the connection pool, you may get some errors. You can refer to this link: https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
And could you please share some more error message ?
I can see that you are creating httpclient connection on each request which possibly be causing this issue. Httpclient creates a socket connection underneath it and has hard limit on it. Even when you dispose it it remains there for couple of mins that can't be used. A good practice is to create single static httpclient connection and reuse it. I am attaching some documents for you to go through.
AzFunction Static HttpClient , Http Client Working , Improper instantiation
We followed this example (http://masstransit-project.com/MassTransit/usage/azure-functions.html) to try to set up Azure Functions as Azure Service Bus event (topic) subscribers using MassTransit (for .Net CORE 2.1, Azure Functions 2.0).
When using Azure Webjobs this is as simple as using RabbitMQ, configure the publisher, let the subscriber configure and set up its queue, and have Masstransit automatically create one topic per event, redirect to queue and to "queue_error" after all retries have failed. You do not have to setup anything manually.
But with Azure Functions we seem to manually (through Service Bus Explorer or ARM templates) have to add the subscribers to the topic (which is created by the publisher on the first event it publishes) and the queues as well (though these don't even seem to be necessary, the events are handled directly by the consuming Azure Function topic subscribers.).
Maybe we are doing something wrong, I cannot see from the docs that MT will not, as it normally does, set up the subscriber andd creating queues when using Azure Functions. But it works, except for when the consumer throws an exception and after all setup retries have been executed. We simply do not get the event in the deadletter queue and the normally MT-generated error queue does not even get generated.
So how do we get MT to create the error queues, and MOVE the failed events there?
Our code:
[FunctionName("OrderShippedConsumer")]
public static Task OrderShippedConsumer(
[ServiceBusTrigger("xyz.events.order/iordershipped", "ordershippedconsumer-queue", Connection = "AzureServiceBus")] Message message,
IBinder binder,
ILogger logger,
CancellationToken cancellationToken,
ExecutionContext context)
{
var config = CreateConfig(context);
var handler = Bus.Factory.CreateBrokeredMessageReceiver(binder, cfg =>
{
var serviceBusEndpoint = Parse.ConnectionString(config["AzureServiceBus"])["Endpoint"];
cfg.CancellationToken = cancellationToken;
cfg.SetLog(logger);
cfg.InputAddress = new Uri($"{serviceBusEndpoint}{QueueName}");
cfg.UseRetry(x => x.Intervals(TimeSpan.FromSeconds(5)));
cfg.Consumer(() => new OrderShippedConsumer(cfg.Log, config));
});
return handler.Handle(message);
}
And the Consumer code:
public OrderShippedConsumer(ILog log, IConfigurationRoot config)
{
this.config = config;
this.log = log;
}
public async Task Consume(ConsumeContext<IOrderShipped> context)
{
// Handle the event
}
}
Is there anyway to get a timestamp as to when a message is placed on my Azure queue? What is the best way?
For example, a partner sends a message to my queue and I want to know the time the partner placed a specific message in the queue.
Thanks
If you're using the .NET API, the property InsertionTime in the CloudQueueMessage you get when fetching a message or peeking the queue will contain;
The time that that message was added to the queue.
For example, in Java code the following would retrieve a message (then get the insertion time) and then peek at the next message (and get its insertion time.) This also shows how access to the queue can be controlled by using a revocable stored access policy string.
public class Dequeue {
public static void main(String[] args) throws InvalidKeyException, URISyntaxException, StorageException
{
String sas = "sv=2012-02-12&sig=XUHWUy2ayrcEjNvZUhcgdPbgKZflwSXxtr0BH87XRII%3D&si=heath";
StorageCredentialsSharedAccessSignature credentials = new StorageCredentialsSharedAccessSignature(sas);
URI baseuri = new URI("http://grassy.queue.core.windows.net");
CloudQueueClient queueClient = new CloudQueueClient(baseuri,credentials);
CloudQueue queue = queueClient.getQueueReference("queue1");
CloudQueueMessage retrievedMessage = queue.retrieveMessage();
System.out.println(retrievedMessage.getMessageContentAsString());
System.out.println(retrievedMessage.getInsertionTime());
queue.deleteMessage(retrievedMessage);
CloudQueueMessage peekedMessage = queue.peekMessage();
System.out.println(peekedMessage.getMessageContentAsString());
System.out.println(peekedMessage.getInsertionTime());
} }