I have this code to put message on the queue
public class AzureQueueService : IAzureQueueService
{
private readonly IConfiguration _config;
private static string azureServiceBusString = null;
public AzureQueueService(IConfiguration config)
{
_config = config;
azureServiceBusString = "connString";
}
public async Task SendMessageAsync<T>(T serviceBusMessage, string queueName)
{
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var queueClient = new ServiceBusClient(azureServiceBusString);
// create the sender
ServiceBusSender sender = queueClient.CreateSender(queueName);
string messageBody = JsonSerializer.Serialize(serviceBusMessage);
var message = new ServiceBusMessage(Encoding.UTF8.GetBytes(messageBody));
// send the message
await sender.SendMessageAsync(message);
}
}
In my startup file I am doing this
services.AddTransient<IAzureQueueService, Core.Services.AzureQueueService>();
And I am using it like this
var queue = new AzureQueueService(config);
await queue.SendMessageAsync(message, "emailqueue");
Could this lead to memory leakage? I mean should I instantiate the ServiceBusClient in the constructor?
Yes, I'd be looking to set up dependency injection for the ServiceBusClient. As per the docs here (for 7.8.x)
The ServiceBusClient, senders, receivers, and processors are safe to cache and use as a singleton for the lifetime of the application, which is best practice when messages are being sent or received regularly. They are responsible for efficient management of network, CPU, and memory use, working to keep usage low during periods of inactivity.
Also, see Best Practises for performance improvements using Service Bus Messaging
Related
I'm trying to update this function to make use of Azure.Messaging.ServiceBus and drop Microsoft.Azure.ServiceBus altogether, however cant seem to find any resources for this. Anybody knows how to send a message to a topic using this package?
The older function is:
public async Task SendMessageToServiceBusTopic<T>(T request, string topicSubName, string submissionNumber)
{
ServiceBusConnectionStringBuilder serviceBusConnectionStringBuilder =
new ServiceBusConnectionStringBuilder(settings.ServiceBusConnectionString)
{
EntityPath = settings.ServiceBusTopic
};
TopicClient topicClient = new TopicClient(serviceBusConnectionStringBuilder);
byte[] bytes = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(request));
await topicClient.SendAsync(new Message(bytes)
{
CorrelationId = context.CorrelationId,
Label=topicSubName,
UserProperties = { new KeyValuePair<string, object>("TrackingId", submissionNumber) }
});
}
So far I have managed:
Am i headed in the right direction?
public async Task SendMessageToServiceBusTopic<T>(T request, string topicSubName, string submissionNumber)
{
ServiceBusClient client = new ServiceBusClient(settings.ServiceBusConnectionString);
ServiceBusSender s = client.CreateSender(settings.ServiceBusTopic);
byte[] bytes = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(request));
await s.SendMessageAsync(new ServiceBusMessage(bytes));
}
While you can construct a Service Bus Client each time, it's not ideal. Assuming you're using the latest In-Proc SDK, you can use one of these options:
[FunctionName("PublishToTopic")]
public static async Task Run(
[TimerTrigger("0 */5 * * * *")] TimerInfo myTimer,
[ServiceBus("<topic_name>", Connection = "<connection_name>")] IAsyncCollector<ServiceBusMessage> collector)
{
await collector.AddAsync(new ServiceBusMessage(new BinaryData($"Message 1 added at: {DateTime.Now}")));
await collector.AddAsync(new ServiceBusMessage(new BinaryData($"Message 2 added at: {DateTime.Now}")));
}
Alternatively,
[FunctionName("PublishWithSender"]
public static async Task Run(
[TimerTrigger("0 */5 * * * *")] TimerInfo myTimer,
[ServiceBus("<topic_name>", Connection = "<connection_name>")] ServiceBusSender sender)
{
await sender.SendMessagesAsync(new[]
{
new ServiceBusMessage(new BinaryData($"Message 1 added at: {DateTime.Now}")),
new ServiceBusMessage(new BinaryData($"Message 2 added at: {DateTime.Now}"))
});
}
For Isolated Worker SDK it's somewhat different. See this post for details.
you're in the right direction. There's no easy migration tool/sample as they are different libraries dealing with the same service (Azure Service Bus).
I was in the same situation like you (trying to migrate to Azure.Messaging.ServiceBus which is now the recommended NuGet from Microsoft).
You can have a look at the migration guide from their Github to have an idea of how to make the transition smoother:
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Azure.Messaging.ServiceBus/MigrationGuide.md
/!\ IQueueClient and ITopicClient (both coming from Microsoft.Azure.ServiceBus) have been merged into a single object called ServiceBusSender.
Why ? to make our life easier ! Indeed, under the hood, ServiceBusSender now takes care of the process of sending the message, especially since a queue and a topic cannot have the same name if I'm not wrong.
Here's a sample code of mine to send a message using the new library:
/// <summary>
/// Send a message by serializing as JSON the object in input.
/// </summary>
/// <param name="queueOrTopicName">Name of the queue or topic where the message will be sent.</param>
/// <param name="messageToSend">Any C# object</param>
/// <returns></returns>
public async Task SendMessage(string queueOrTopicName, object messageToSend)
{
//ServiceBusSender should not be disposed (according to the documentation, Github, etc.)
using ServiceBusClient client = new ServiceBusClient(connectionString: "Your-ServiceBus-ConnectionString");
using ServiceBusSender sender = client.CreateSender(_queueOrTopicName);
// create a message that we can send. UTF-8 encoding is used when providing a string.
ServiceBusMessage message = BuildServiceBusMessage(messageToSend);
// Finally send the message
await sender.SendMessageAsync(message);
}
private ServiceBusMessage BuildServiceBusMessage<T>(T entity)
{
string serializedMessage = JsonConvert.SerializeObject(entity); // Still using Newtonsoft.Json but I don't see any obstacles of using System.Text.Json.
ServiceBusMessage message = new ServiceBusMessage(serializedMessage)
{
MessageId = Guid.NewGuid().ToString(),
ContentType = "application/json; charset=utf-8",
};
return message;
}
If you have any needs to do Dependency Injection (to re-use the same ServiceBusClient object and avoiding instanciating ServiceBusClient for each message you want to send for example), you can refer to this stackoverflow that I discovered this week
How to register ServiceBusClient for dependency injection?
While converting my project from .Net framework 4.7 to .Net core 2.1, I'm facing issue with Servicebus MessagingFactory. I don't see any MessagingFactory class in new nuget package Microsoft.Azure.ServiceBus for .Net core.
My .Net framework 4.7 Code
private static readonly string messagingConnectionString = Environment.GetEnvironmentVariable("ServiceBusConnection");
private static Lazy<MessagingFactory> lazyMessagingFactory = new Lazy<MessagingFactory>(() =>
{
return MessagingFactory.CreateFromConnectionString(messagingConnectionString);
});
public static MessagingFactory MessagingFactory
{
get
{
return lazyMessagingFactory.Value;
}
}
public static MessagingFactory EventHubMessageFactory
{
get
{
return lazyEventhubMessagingFactory.Value;
}
}
public async Task SendMessageToQueueAsync(string queueName, string message)
{
QueueClient queueClient = MessagingFactory.CreateQueueClient(queueName);
BrokeredMessage brokeredMessage = new BrokeredMessage(new MemoryStream(Encoding.UTF8.GetBytes(message)), true);
await queueClient.SendAsync(brokeredMessage);
}
It was a best practices for high performance application, Also I have many queues under single service bus namespace and I push message based on configuration. I don't want to create QueueClient object in every request and don't want to maintain connection string for every queue.
What is alternate of MessagingFactory in .Net Core?
There are major changes when you are migrating .NetFramework code into .Netcore, you can see Guide for migrating to Azure.Messaging.ServiceBus from Microsoft.Azure.ServiceBus
Example below
static void Main(string[] args)
{
string connectionString = "<connection_string>";
string queueName = "<queue_name>";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
var client = new ServiceBusClient(connectionString);
// create the sender
ServiceBusSender sender = client.CreateSender(queueName);
for (int i = 0; i < 10; i++)
{
// create a message that we can send. UTF-8 encoding is used when providing a string.
ServiceBusMessage message = new ServiceBusMessage($"Hello world {i}!");
// send the message
sender.SendMessageAsync(message).GetAwaiter().GetResult();
}
sender.DisposeAsync().GetAwaiter().GetResult();
client.DisposeAsync().GetAwaiter().GetResult();
}
https://github.com/MicrosoftDocs/azure-docs/issues/46830
https://github.com/Azure/azure-service-bus-dotnet/issues/556
While MessagingFactory is gone, the idea of connection pooling and sharing connections is there. When you create your clients, passing a connection will reuse it. When passing a connection string, will cause clients to establish a new connection.
So you can manually create a ServiceBusConnection or reuse one of an existing client. You can pass the connection object in the constructors of the clients you create. Take care not to close a connection accidentally, e.g. by closing the client that created it.
Problem Statement: We have a requirement to upload log data to Azure Storage from a Xamarin.IOS application. The logs are not created by the user of the application, and there's no constraint on the user to keep the application open for any amount of time after the logs are generated. We want to reliably upload our logs with a couple points in mind:
The user might send the app into the background
The file sizes can be up to 15MB
We don't care when we get them. We're open to scheduling a task for this.
In looking at potential solutions to this problem, the Xamarin documentation states that in iOS7+:
NSURLSession allows us to create tasks to:
Transfer content through network and device interruptions.
Upload and download large files ( Background Transfer Service ).
So it seems like NSURLSession is a good candidate for this sort of work, but I wonder if I am reinventing the wheel. Does the WindowsAzure.Storage client library respect app backgrounding with an upload implementation based on NSURLSession, or if I want to upload the data in the background, is it necessary to upload to an intermediate server I control with a POST method, and then relay data to Azure Storage? There doesn't seem to be any indication from the public Azure documentation that uploads can be done via scheduled task.
I got this working. I've simplified the classes and methods into a single method. Only the necessities are here.
public void UploadFile(File playbackFile)
{
/// Specify your credentials
var sasURL = "?<the sastoken>";
/// Azure blob storage URL
var storageAccount = "https://<yourstorageaccount>.blob.core.windows.net/<your container name>";
/// specify a UNIQUE session name
var configuration =
NSUrlSessionConfiguration.CreateBackgroundSessionConfiguration("A background session name");
/// create the session with a delegate to recieve callbacks and debug
var session = NSUrlSession.FromConfiguration(
configuration,
new YourSessionDelegate(),
new NSOperationQueue());
/// Construct the blob endpoint
var url = $"{storageAccount}/{playbackFile.Name}{sasURL}";
var uploadUrl = NSUrl.FromString(url);
/// Add any headers for Blob PUT. x-ms-blob-type is REQUIRED
var dic = new NSMutableDictionary();
dic.Add(new NSString("x-ms-blob-type"), new NSString("BlockBlob"));
/// Create the request with NSMutableUrlRequest
/// A default NSUrlRequest.FromURL() is immutable with a GET method
var request = new NSMutableUrlRequest(uploadUrl);
request.Headers = dic;
request.HttpMethod = "PUT";
/// Create the task
var uploadTask = session.CreateUploadTask(
request,
NSUrl.FromFilename(playbackFile.FullName));
/// Start the task
uploadTask.Resume();
}
/// Delegate to recieve callbacks. Implementations are omitted for brevity
public class YourSessionDelegate: NSUrlSessionDataDelegate
{
public override void DidBecomeInvalid(NSUrlSession session, NSError error)
{
Console.WriteLine(error.Description);
}
public override void DidSendBodyData(NSUrlSession session, NSUrlSessionTask task, long bytesSent, long totalBytesSent, long totalBytesExpectedToSend)
{
Console.WriteLine(bytesSent);
}
public override void DidReceiveData(NSUrlSession session, NSUrlSessionDataTask dataTask, NSData data)
{
Console.WriteLine(data);
}
public override void DidCompleteWithError(NSUrlSession session, NSUrlSessionTask task, NSError error)
{
var uploadTask = task as NSUrlSessionUploadTask;
Console.WriteLine(error?.Description);
}
public override void DidReceiveResponse(NSUrlSession session, NSUrlSessionDataTask dataTask, NSUrlResponse response, Action<NSUrlSessionResponseDisposition> completionHandler)
{
Console.WriteLine(response);
}
public override void DidFinishEventsForBackgroundSession(NSUrlSession session)
{
using (AppDelegate appDelegate = UIApplication.SharedApplication.Delegate as AppDelegate)
{
var handler = appDelegate.BackgroundSessionCompletionHandler;
if (handler != null)
{
appDelegate.BackgroundSessionCompletionHandler = null;
handler();
}
}
}
}
Helpful documentation:
https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob
https://developer.apple.com/documentation/foundation/nsmutableurlrequest/1408793-setvalue
https://learn.microsoft.com/en-us/dotnet/api/foundation.insurlsessiontaskdelegate?view=xamarin-ios-sdk-12
Hopefully someone finds this useful and spends less time on this than I did. Thanks #SushiHangover for pointing me in the right direction.
I got into an issue with IMessageSessionAsyncHandlerFactory where new instances of IMessageSessionAsyncHandler are not created when the volume of writing goes to 0 and then up to a normal level.
To be more precise, I'm using SessionHandlerOptions with a value of 500 for MaxConcurrentSessions. This allows reading at a speed of more than 1k msg/s.
The queue I'm reading from is a partitioned queue.
The volume of messages in the queue is rather constant, but from time to time it gets down to 0. When the volume gets back to the normal level, the SessionFactory is not spawning any handlers so I'm not able to read messages anymore. It's like the sessions were not correctly recycled or are held into a sort of continuous waiting.
Here is the code for the factory registering:
private void RegisterHandler()
{
var sessionHandlerOptions = new SessionHandlerOptions
{
AutoRenewTimeout = TimeSpan.FromMinutes(1),
MessageWaitTimeout = TimeSpan.FromSeconds(1),
MaxConcurrentSessions = 500
};
_queueClient.RegisterSessionHandlerFactoryAsync(new SessionHandlerFactory(_callback), sessionHandlerOptions);
}
The factory class:
public class SessionHandlerFactory : IMessageSessionAsyncHandlerFactory
{
private readonly Action<BrokeredMessage> _callback;
public SessionHandlerFactory(Action<BrokeredMessage> callback)
{
_callback = callback;
}
public IMessageSessionAsyncHandler CreateInstance(MessageSession session, BrokeredMessage message)
{
return new SessionHandler(session.SessionId, _callback);
}
public void DisposeInstance(IMessageSessionAsyncHandler handler)
{
var disposable = handler as IDisposable;
disposable?.Dispose();
}
}
And the handler:
public class SessionHandler : MessageSessionAsyncHandler
{
private readonly Action<BrokeredMessage> _callback;
public SessionHandler(string sessionId, Action<BrokeredMessage> callback)
{
SessionId = sessionId;
_callback = callback;
}
public string SessionId { get; }
protected override async Task OnMessageAsync(MessageSession session, BrokeredMessage message)
{
try
{
_callback(message);
}
catch (Exception ex)
{
Logger.Error(...);
}
}
I can see that the session handlers are closed and that the factories are disposed when the writing/reading is at a normal level. However, once the queue empties, there's no way new session handlers are created. Is there a policy for allocating session IDs that forbids reallocating the same sessions after a period of inactivity?
Edit 1:
I'm adding two pictures to illustrate the behavior:
When the writer is stopped and restarted, the running reader is not able to read as much as before.
The number of sessions created after that moment is also much lower than before:
The volume of messages in the queue is rather constant, but from time to time it gets down to 0. When the volume gets back to the normal level, the SessionFactory is not spawning any handlers so I'm not able to read messages anymore. It's like the sessions were not correctly recycled or are held into a sort of continuous waiting.
When using IMessageSessionHandlerFactory to control how the IMessageSessionAsyncHandler instances are created, you could try to log the creation and destruction for all of your IMessageSessionAsyncHandler instances.
Based on your code, I created a console application to this issue on my side. Here is my code snippet for initializing queue client and handling messages:
InitializeReceiver
static void InitializeReceiver(string connectionString, string queuePath)
{
_queueClient = QueueClient.CreateFromConnectionString(connectionString, queuePath, ReceiveMode.PeekLock);
var sessionHandlerOptions = new SessionHandlerOptions
{
AutoRenewTimeout = TimeSpan.FromMinutes(1),
MessageWaitTimeout = TimeSpan.FromSeconds(5),
MaxConcurrentSessions = 500
};
_queueClient.RegisterSessionHandlerFactoryAsync(new SessionHandlerFactory(OnMessageHandler), sessionHandlerOptions);
}
OnMessageHandler
static void OnMessageHandler(BrokeredMessage message)
{
var body = message.GetBody<Stream>();
dynamic recipeStep = JsonConvert.DeserializeObject(new StreamReader(body, true).ReadToEnd());
lock (Console.Out)
{
Console.ForegroundColor = ConsoleColor.Cyan;
Console.WriteLine(
"Message received: \n\tSessionId = {0}, \n\tMessageId = {1}, \n\tSequenceNumber = {2}," +
"\n\tContent: [ title = {3} ]",
message.SessionId,
message.MessageId,
message.SequenceNumber,
recipeStep.title);
Console.ResetColor();
}
Task.Delay(TimeSpan.FromSeconds(3)).Wait();
message.Complete();
}
Per my test, the SessionHandler could work as expected when the volume of messages in the queue from normal to zero and from zero to normal for some time as follows:
I also tried to leverage QueueClient.RegisterSessionHandlerAsync to test this issue and it works as well. Additionally, I found this git sample about Service Bus Sessions, you could refer to it.
I have been using the ServiceStack MQ Server/Client to empower a message based architecture in my platform and it has been working flawlessly. I am now trying to do something that I do not believe is supported by the SS Message Producer/Consumer.
Essentially I am firing off messages (events) at a centralized data center and I have ~2000 decentralized nodes all over the US over a non reliable network that need to potentially know about about this event BUT the event needs to be targeted to only one of the ~2000 nodes. I need the flexibility of the arbitrarily named channels with Pub/Sub but the durability of the MQ. I started off with Pub/Sub but the network is too unreliable so I have moved the solution to use the RedisMQServer. I have it working but wanted to make sure I am not missing something in the interface. I am curious if the creators of SS have thought through this use case and if so what the outcome of that discussion was? This does fight the concept of using the POCO's to drive the outcomes/actions of the message consumption. Maybe that is the reason?
Here is my producer
public ExpressLightServiceResponse Get(ExpressLightServiceRequest query)
{
var result = new ExpressLightServiceResponse();
var assemblyBuilder = Thread.GetDomain().DefineDynamicAssembly(new AssemblyName("ArbitaryNamespace"), AssemblyBuilderAccess.Run);
var moduleBuilder = assemblyBuilder.DefineDynamicModule("ModuleName");
var typeBuilder = moduleBuilder.DefineType(string.Format("EventA{0}", query.Store), TypeAttributes.Public);
typeBuilder.DefineDefaultConstructor(MethodAttributes.Public);
var newType = typeBuilder.CreateType();
using (var messageProducer = _messageService.CreateMessageProducer())
{
var message = MessageFactory.Create(newType.CreateInstance());
messageProducer.Publish(message);
}
return result;
}
Here is my consumer
public class ServerAppHost : AppHostHttpListenerBase
{
private readonly string _store;
public string StoreQueue => $"EventA{_store}";
public ServerAppHost(string store) : base("Express Light Server", typeof(PubSubServiceStatsService).Assembly)
{
_store = store;
}
public override void Configure(Container container)
{
container.Register<IRedisClientsManager>(new PooledRedisClientManager(ConfigurationManager.ConnectionStrings["Redis"].ConnectionString));
var assemblyBuilder = Thread.GetDomain().DefineDynamicAssembly(new AssemblyName("ArbitaryNamespace"), AssemblyBuilderAccess.Run);
var moduleBuilder = assemblyBuilder.DefineDynamicModule("ModuleName");
var typeBuilder = moduleBuilder.DefineType(StoreQueue, TypeAttributes.Public);
typeBuilder.DefineDefaultConstructor(MethodAttributes.Public);
var newType = typeBuilder.CreateType();
var mi = typeof(Temp).GetMethod("Foo");
var fooRef = mi.MakeGenericMethod(newType);
fooRef.Invoke(new Temp(container.Resolve<IRedisClientsManager>()), null);
}
}
public class Temp
{
private readonly IRedisClientsManager _redisClientsManager;
public Temp(IRedisClientsManager redisClientsManager)
{
_redisClientsManager = redisClientsManager;
}
public void Foo<T>()
{
var mqService = new RedisMqServer(_redisClientsManager);
mqService.RegisterHandler<T>(DoWork);
mqService.Start();
}
private object DoWork<T>(IMessage<T> arg)
{
//Do work
return null;
}
}
What this gives me is the flexibility of Pub/Sub with the durability of a Queue. Does anyone see/know of a more "native" way to achieve this?
There should only be 1 MQ Host registered in your AppHost so I'd firstly remove it out of your wrapper class and have it just register the handler, e.g:
public override void Configure(Container container)
{
//...
container.Register<IMessageService>(
c => new RedisMqServer(c.Resolve<IRedisClientsManager>());
var mqServer = container.Resolve<IMessageService>();
fooRef.Invoke(new Temp(mqServer), null);
mqServer.Start();
}
public class Temp
{
private readonly IMessageService mqServer;
public Temp(IMessageService mqServer)
{
this.mqServer = mqServer;
}
public void Foo<T>() => mqService.RegisterHandler<T>(DoWork);
}
But this approach isn't good fit for ServiceStack which encourages the use of code-first Messages which defines the Service Contract that client/servers use to process the messages that are sent and received. So if you want to use ServiceStack for sending custom messages I'd recommend either having a separate class per message otherwise have a generic Type like SendEvent where the message or event type is a property on the class.
Otherwise if you want to continue with custom messages don't use RedisMqServer, you can just use a dedicated MQ like Rabbit MQ or if you prefer use a Redis List directly - which is the data structure that all Redis MQ's use underneath.