azure service bus message Id us email ID - azure

I am creating a batch job to read email from an inbox using Microsoft Graph API and publish the content of the email using service bus.
I put a duplicate detection in my azure service bus based on the assigned Message ID. To prevent publishing the same email, I put the Email message Id to the Azure Service Bus Message ID.
However it seems that the length of the email message ID is too long and I am unable to fit it as the azure service bus message ID. Any solution on how to put the email message ID as the Azure Service Bus message ID? (I am not allowed to store the email message ID in any persistence storage, I can only publish it via Azure Service Bus)
Thanks

The length of the service bus message ID can't exceed 128 characters. Generally, the value of
email ids that you retrieved from Microsoft Graph API look like:
AAMkADU1NzNkY2I2LWIxY2QtNGFhOC1iYmE5LWEzODA2ZjJkMjQxNwBGAAAAAABRRlt89urpT62rDl3o5R1RBwCSQ84srYEXTY2slhDtcrhrAAAAAAEMAACSQ84srYEXTY2slhDtcrhrAABlQp2FAAA=
Generally, its length is 188 characters which caused this issue.
To solve this issue, you can try to use MD5 to create a unified identifier for the email ID, by default, its length will be 32 characters, though this value has the possibility to collision with a hash that from a different string, but its possibility is low(about 2^-128).
If you are using C#, just try the code below:
using Azure.Messaging.ServiceBus;
using System;
using System.Security.Cryptography;
using System.Text;
namespace serviceBusTest
{
class Program
{
static void Main(string[] args)
{
string connectionString = "";
string queueName = "";
var client = new ServiceBusClient(connectionString);
ServiceBusSender sender = client.CreateSender(queueName);
ServiceBusMessage message = new ServiceBusMessage();
var emailID = "AAMkADU1NzNkY2I2LWIxY2QtNGFhOC1iYmE5LWEzODA2ZjJkMjQxNwBGAAAAAABRRlt89urpT62rDl3o5R1RBwCSQ84srYEXTY2slhDtcrhrAAAAAAEMAACSQ84srYEXTY2slhDtcrhrAABlQp2FAAA=";
message.MessageId = GetMd5Hash(MD5.Create(), emailID);
sender.SendMessageAsync(message).GetAwaiter().GetResult();
ServiceBusReceiver receiver = client.CreateReceiver(queueName);
ServiceBusReceivedMessage receivedMessage = receiver.ReceiveMessageAsync().GetAwaiter().GetResult();
Console.WriteLine(receivedMessage.MessageId);
}
static string GetMd5Hash(MD5 md5Hash, string input)
{
byte[] data = md5Hash.ComputeHash(Encoding.UTF8.GetBytes(input));
StringBuilder sBuilder = new StringBuilder();
for (int i = 0; i < data.Length; i++)
{
sBuilder.Append(data[i].ToString("x2"));
}
return sBuilder.ToString();
}
}
}
Result:

Related

Azure service bus - how to add metadata to the message

I am using .net core web app as the publisher and .net core console app as subscriber.
I am able to successfully pass messages between these two systems using Managed Identities - set up in Azure portal.
My question is I need to add metadata to the the message that is being sent. How do I do that ?
Below is my publisher code :
string data = JsonConvert.SerializeObject(payloadEvents);
Message message = new Message(Encoding.UTF8.GetBytes(data));
var tokenProvider = TokenProvider.CreateManagedIdentityTokenProvider();
TopicClient sendClient = new TopicClient(_serviceBusNamespace, _topicName, tokenProvider, retryPolicy: null);
await sendClient.SendAsync(message);
Message object has a property called UserProperties that can be used to set custom metadata for that message.
Something like:
string data = JsonConvert.SerializeObject(payloadEvents);
Message message = new Message(Encoding.UTF8.GetBytes(data));
message.UserProperties.Add("key1", "value1");
message.UserProperties.Add("key2", "value2");
var tokenProvider = TokenProvider.CreateManagedIdentityTokenProvider();
TopicClient sendClient = new TopicClient(_serviceBusNamespace, _topicName, tokenProvider, retryPolicy: null);
await sendClient.SendAsync(message);

Sending message to a specific group of subscriber in azure service bus topic with masstransit

I'm new to azure service bus and masstransit. I'm looking for a solution to a specific situation.
I have a azure service bus topic with multiple subscribers. Subscriber will receive message based on filters. I've created the topic and subscriber with code below
class Program
{
static void Main(string[] args)
{
string connectionString = "Endpoint connection string";
// the names of topics and subscriptions we'll be working with
const string topicName = "MyTestTopic";
const string allMessagesSubName = "AllMessages";
const string filteredSubName1 = "Filtered1";
const string filteredSubName2 = "Filtered2";
// let's create the topic if it doesn't already exist...
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
if (!namespaceManager.TopicExists(topicName))
{
var td = new TopicDescription(topicName);
namespaceManager.CreateTopic(td.Path);
}
if (!namespaceManager.SubscriptionExists(topicName, allMessagesSubName))
{
namespaceManager.CreateSubscription(topicName, allMessagesSubName);
}
if (!namespaceManager.SubscriptionExists(topicName, filteredSubName1))
{
namespaceManager.CreateSubscription(
new SubscriptionDescription(topicName, filteredSubName1),
new Microsoft.ServiceBus.Messaging.SqlFilter("From LIKE '%Smith'"));
}
if (!namespaceManager.SubscriptionExists(topicName, filteredSubName2))
{
namespaceManager.CreateSubscription(
new SubscriptionDescription(topicName, filteredSubName2),
new Microsoft.ServiceBus.Messaging.SqlFilter("sys.Label='important'"));
}
var message1 = new BrokeredMessage("Hello World");
var message2 = new BrokeredMessage("Second message");
message2.Label = "important";
var message3 = new BrokeredMessage("Third message");
message3.Properties["From"] = "Kelly Smith";
message3.Label = "information";
var client = TopicClient.CreateFromConnectionString(connectionString, topicName);
client.Send(message1);
client.Send(message2);
client.Send(message3);
client.Close();
}
}
Here in the code we're adding Message custom properties like From.
Now I want to send such message using masstransit. In masstransit I cannot find any option of adding Message custom properties using the Publish() method. Is there any way that I can send these messages using masstransit where these filters can be used?
NB: I've read the answer of this question But the anwer here tells us to filter the messages in subscriber side. What I want is that this filtering will occur before reaching the subscriber.
When using Azure Service Bus with MassTransit, you can add subscription endpoints in additional to regular endpoints. When configuring a subscription endpoint, you should be able to specify rules and/or filters as part of the subscription. Which is exactly what you're doing above, so that is handled.
The other part, adding properties to the message, can be done by adding text headers to the SendContext. Those headers are copied to the message Properties collection, which I believe can be used to filter messages using a "SQL" filter (which is configured on the subscription endpoint, or the topic subscription on a receive endpoint).

Configuring Twilio SMS from Azure Functions v2

I have some code where I'm reading messages off of an Azure Event Hub that I want to either send an email or send an SMS.
The email is working through send grid, but I'm not sure how to configure the SMS part though.
I think I'd want to use Twilio and here's a sample of what my code's like. The "messageCollector" works for sending Email since there's some configuration for SendGrid in the local json. How do I configure Twilio?
[FunctionName("SendAlert")]
public static async Task Run(
[EventHubTrigger("v1-email-hub", Connection = "EventHubConnection")] EventData[] events,
[SendGrid] IAsyncCollector<SendGridMessage> messageCollector,
[TwilioSms] IAsyncCollector<CreateMessageOptions> smsCollector,
[Inject] NotificationEventLogic eventLogic,
ILogger log)
{
foreach (EventData eventData in events)
{
string messageBody = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
var notificationEvents = JsonConvert.DeserializeObject<List<NotificationEvent>>(messageBody);
foreach (var ev in notificationEvents)
{
if (ev.NotificationEventType == NotificationEventType.Email)
{
var message = new SendGridMessage();
// ... ... make message and add it
await messageCollector.AddAsync(message);
}
else if (ev.NotificationEventType == NotificationEventType.SMS)
{
// Not sure how to get this to work
var mobileMessage = new CreateMessageOptions(new PhoneNumber(ev.Data))
{
Body = $"Notification {ev.NotificationId}"
};
await smsCollector.AddAsync(mobileMessage);
}
// await smsCollector.AddAsync()
await eventLogic.CreateEventAsync(ev);
}
}
}
You will need to configure it in attribute
[TwilioSms(AccountSidSetting = "TwilioAccountSid", AuthTokenSetting = "TwilioAuthToken", From = "+1425XXXXXXX")]
as it mentioned in documentation
TwilioAccountSid This value must be set
to the name of an app setting that holds your Twilio Account Sid e.g.
TwilioAccountSid. If not set, the default app setting name is
"AzureWebJobsTwilioAccountSid".
TwilioAuthToken This value must be set to
the name of an app setting that holds your Twilio authentication token
e.g. TwilioAccountAuthToken. If not set, the default app setting name
is "AzureWebJobsTwilioAuthToken".

Windows Azure worker role and SendGrid "Bad Key Path!" Error

I'm trying to use SendGrid to send an email from an Azure worker role every time there are certain exceptions, but I can't get the email to send. I am using SendGridMail version 6.1.0.0 and SendGrid.SmtpApi version 1.3.1.0 which I installed via nuget and .Net 4.5. I am currently debugging locally with plans to deploy to Azure if i can get the emails to successfully send.
SendGridMessage myMessage = new SendGridMessage();
List<String> recipients = new List<String> { #"John Doe <johnd#outlook.com>", #"Peter Howe <perterhowe#gmail.com>" };
myMessage.AddTo(recipients);
myMessage.From = new MailAddress("myemail#test.com");
myMessage.Subject = "Error in Update";
myMessage.Text = "TESTING 123";
string username = XXXXXX;
string password = XXXXXXX;
// Create credentials, specifying your user name and password.
var credentials = new NetworkCredential(username, password);
// Create an Web transport for sending email.
var transportWeb = new Web(credentials);
// Send the email.
await transportWeb.DeliverAsync(myMessage);
As far as I can see I'm not getting any errors except when I debug and look at myMessage the Header has an error.
When I tried initializing a new empty header (var header = new Header();) I noticed there were still errors on that
To = 'header.To' threw an exception of type 'System.ArgumentException' Message = "Bad key path!"
Does anyone know what this means? Or if this could be causing the emails not to send?
The answer to your other question actually uses SendGrid:
Alerts for exceptions in an Azure worker role
There are three globalvariables:
public const string SmtpServerHost = "smtp.sendgrid.net";
public const string SmtpServerUserName = "[useridfromsendgrid#azure.com]";
public const string SmtpServerPassword = "[password from sendgrid]";
You actually do not need to use the SDK, just setup the account in Azure portal, and save your creds in your project.
You can send emails locally, but if you are on a work network, the firewall may block the emails from being sent. The code I posted I placed in an email service in my namespace.
It has be deployed to Azure to work. It won't work locally.

Azure service bus dropping messages?

I am trying to build a simple Web API REST service in Azure with a service bus queue worker on the back end. I can send a single message from the Web API to the worker just fine. However, I was trying to send more messages just to see how everything works. So, I created a simple controller that looks like this:
for (int i = 0; i < 100; i++)
{
var msg = new BrokeredMessage("Ping");
BioConnector.QueueConnector.OrdersQueueClient.Send(msg);
}
When I call the controller, I am only getting about 1/2 or so of the messages being received by the worker. The rest seem to be dropped.
I had issues with getting only about half the messages using the sample code posted here, so I wrote my own test code. I've tried it with > 100 queue messages and have always had 100% send/rec'd parity. Perhaps you had a similar issue with the code.
Create a new C# console project.
Add a reference to the Microsoft.ServiceBus assembly located in C:\Program Files\Microsoft SDKs\Windows Azure.NET SDK\2012-06\ref\Microsoft.ServiceBus.dll.
In the app.config, change it to this with your own values provided:
<appSettings>
<add key="Microsoft.ServiceBus.ConnectionString" value="Endpoint=sb://blahblah.servicebus.windows.net/;SharedSecretIssuer=owner;SharedSecretValue=pDk0b....=" />
</appSettings>
Add these using directives:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
using System.Xml.Linq;
using Microsoft.ServiceBus;
using Microsoft.ServiceBus.Messaging;
using System.Configuration;
using System.Threading;
Change the code method to the following:
class Program
{
static void Main(string[] args)
{
string connectionString = ConfigurationSettings.AppSettings["Microsoft.ServiceBus.ConnectionString"];
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
QueueDescription queueDesc = new QueueDescription("TestQueue");
if (!namespaceManager.QueueExists(queueDesc.Path))
{
namespaceManager.CreateQueue(queueDesc);
}
QueueClient topicClient = QueueClient.CreateFromConnectionString(connectionString, queueDesc.Path);
int sentMsgCount = 0;
int recdMsgCount = 0;
for (int i = 0; i < 100; i++)
{
BrokeredMessage msg = new BrokeredMessage("Test message " + i);
topicClient.Send(msg);
sentMsgCount++;
Console.WriteLine("Sent Message: " + msg);
}
QueueClient subClient = QueueClient.CreateFromConnectionString(connectionString, queueDesc.Path);
bool moreMessages = true;
while (moreMessages)
{
BrokeredMessage recdMsg = subClient.Receive(TimeSpan.FromSeconds(3));
if (recdMsg != null)
{
Console.WriteLine("Received Message: " + recdMsg);
recdMsgCount++;
recdMsg.Complete();
}
else
{
moreMessages = false;
}
}
Console.WriteLine("# of sent msgs: " + sentMsgCount + ", # of rec'd msgs: " + recdMsgCount);
Console.Read();
}
}
This was a weird problem. Through a random walk through "trying things" I ended up changing the string name of the queue and then everything started working again. I didn't change anything but the name of the queue - no changes to any configuration parameters at all.
It appears to be something buggy with that particular queue on Azure.
Azure Service Bus provides durable messaging so you will not loose any messages. Some items to investigate further:
1) Is there another instance of the worker role that is pulling off messages from that queue
2) are you using peek-lock as the receive mode as that will be the only way to guarantee at least once delivery. Receive and delete mode does not have the guarantee
3) are the messages going into dead-lettered queue due to either message expiry or exceeding max delivery count, I.e. They are received but not completed several times
4) if none of the above apply then raise a support ticket and the Azure product team can investigate the symptoms because as I mentioned this is a durable messaging system so no messages will be "lost".
I had the WindowsAzure.ServiceBus NuGet package in my project and used QueueClient.Send() to send messages and faced the same messages lost issue.
My solution to completely solve the issue:
On the send side, I had to use REST API to send messages.
On the receive side, this is how I extracted the message body:
using (var stream = brokeredMessage.GetBody<Stream>())
{
using (var streamReader = new StreamReader(stream, Encoding.UTF8))
{
var msg = streamReader.ReadToEnd();
// Convert the JSON message to an object
// var obj = JsonConvert.DeserializeObject<ObjectType>(msg);
}
}

Resources