I have many storage accounts that are created programmatically, and I'd like to create an event subscription in each to listen to BlobCreated events with an existing Azure Function endpoint. Right now, I'm doing this manually by going to the portal, but it's been time-consuming to say the least.
Are there any C# code examples that would create an event subscription using Azure credentials?
Please find an example in GitHub
static async Task CreateEventGridEventSubscriptionAsync(string azureSubscriptionId, string eventSubscriptionName, EventGridManagementClient eventGridMgmtClient)
{
Console.WriteLine($"Creating an event subscription to Azure subscription {azureSubscriptionId} with destination as queue {QueueName} under storage account {StorageAccountId}");
string scope = $"/subscriptions/{azureSubscriptionId}";
EventSubscription eventSubscription = new EventSubscription()
{
Destination = new StorageQueueEventSubscriptionDestination()
{
ResourceId = StorageAccountId,
QueueName = QueueName
},
// The below are all optional settings
EventDeliverySchema = EventDeliverySchema.EventGridSchema,
Filter = new EventSubscriptionFilter()
{
IsSubjectCaseSensitive = false,
SubjectBeginsWith = "",
SubjectEndsWith = ""
}
};
EventSubscription createdEventSubscription = await eventGridMgmtClient.EventSubscriptions.CreateOrUpdateAsync(scope, eventSubscriptionName, eventSubscription);
Console.WriteLine("EventGrid event subscription created with name " + createdEventSubscription.Name);
}
Please find the official documentation in MS Docs - IEventSubscriptionsOperations Interface
MS Docs - EventSubscriptionsOperationsExtensions.CreateOrUpdateAsync(IEventSubscriptionsOperations, String, String, EventSubscription, CancellationToken) Method
Related
Given a Event Hub Name, how can I get connection string in C#?
I googled a bit, but nothing useful found so far.
Thanks
Using AAD authentication for an EventHub
var credential = new DefaultAzureCredential();
// or use
// var credential = new Azure.Identity.ClientSecretCredential("tenantId", "clientId", "clientSecret");
EventHubProducerClient producerClient = new EventHubProducerClient(txtNamespace.Text, txtEventHub.Text, credential
var consumerClient = new EventHubConsumerClient(EventHubConsumerClient.DefaultConsumerGroupName, txtNamespace.Text, txtEventHub.Text, credential)
Full example and docs
Acquiring the Connection Strings of configured Access Policies
You can use these two Nuget packages:
Azure.ResourceManager.EventHubs
Azure.Identity
Then you can use the resource group name and the eventhub name to retrieve the connection string. You will need to iterate the subscriptions and resource groups if you don't have this information.
using Azure.Identity;
using Azure.ResourceManager;
using Azure.ResourceManager.EventHubs;
ArmClient client = new ArmClient(new DefaultAzureCredential());
// Or use
// ArmClient client = new ArmClient(new Azure.Identity.ClientSecretCredential("tenantId", "clientId", "clientSecret"));
var subscription = await client.GetDefaultSubscriptionAsync();
var resourceGroup = await subscription.GetResourceGroupAsync("myresourcegroup");
var eventhubNamespace = await resourceGroup.Value.GetEventHubsNamespaceAsync("namespacename");
var rules = eventhubNamespace.Value.GetEventHubsNamespaceAuthorizationRules();
foreach (var rule in rules)
{
var keys = await rule.GetKeysAsync();
Console.WriteLine(keys.Value.PrimaryConnectionString);
Console.WriteLine(keys.Value.SecondaryConnectionString);
}
Not sure if this is what you mean, but if you want to access an Event Hub through C# you need to provide the EH connection string into your code. This can be retrieved by adding a Shared access policy for the Event hub that you are trying to access.
Edit: If you are trying to actually create the connection string yourself you could follow this sample where you create the SAS-token yourself. But you would still need to provide the Primary key that is set on the policy from Azure.
I want to stream the events from different subscription to a single eventhub on azure.
At present I have configured eventhub to a single subscription and events are being streamed. I have a java client which consumes these events and stores it on my persistence layer. My java client looks like..
private void processUsingProcessorClient(){
List<Disposable> subscriptions = null;
try {
EventHubConsumerAsyncClient eventHubConsumerAsyncClient = new EventHubClientBuilder()
.consumerGroup(EventHubClientBuilder.DEFAULT_CONSUMER_GROUP_NAME)
.connectionString(CONNECTION_STRING, EVENT_HUB_NAME)
.credential("*******.servicebus.windows.net","maney-event-hub",createClientSecretCredential())
.buildAsyncConsumerClient();
ReceiveOptions receiveOptions = new ReceiveOptions().setTrackLastEnqueuedEventProperties(true);
List<String> block = eventHubConsumerAsyncClient.getPartitionIds().collectList().block();
Iterator<String> iterator = block.stream().iterator();
String partitionID = null;
subscriptions = new ArrayList<>(block.size());
while(iterator.hasNext()){
partitionID = iterator.next();
Disposable subscription = eventHubConsumerAsyncClient.receiveFromPartition(
partitionID,
EventPosition.fromOffset(0),receiveOptions).subscribe(PARTITION_PROCESSOR,ERROR_HANDLER);
subscriptions.add(subscription);
}
System.in.read();
}catch (Exception ex){
ex.printStackTrace();
} finally {
if(subscriptions != null){
subscriptions.forEach( subscrip -> {
subscrip.dispose();
});
}
}
}
private final Consumer<PartitionEvent> PARTITION_PROCESSOR = partitionEvent -> {
EventData event = partitionEvent.getData();
PartitionContext partitionContext = partitionEvent.getPartitionContext();
String contents = new String(event.getBody(), UTF_8);
LastEnqueuedEventProperties properties = partitionEvent.getLastEnqueuedEventProperties();
System.out.printf("Information received at %s. Last enqueued sequence number: %s%n",properties.getRetrievalTime(), properties.getSequenceNumber());
System.out.printf("Partition[%s] with Offset-[%s] and Sequence Number[%s] has contents: '%s'%n",
partitionContext.getPartitionId(),
event.getOffset(),
event.getSequenceNumber(),
contents);
};
private final Consumer<Throwable> ERROR_HANDLER = errorContext -> {
System.out.printf("Error occurred in partition processor");
errorContext.printStackTrace();
};
public ClientSecretCredential createClientSecretCredential() {
ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
.clientId("****************")
.clientSecret("******************")
.tenantId("**********************")
.build();
return clientSecretCredential;
}
I'm able to read all the events from a single subscription. However I need to do data analytics on these events from different subscriptions too. How do I configre Azure Eventhub to listen to multiple subscriptions ?
I read on Stackoverflow suggestions about creating consumer groups to solve this issue, however I'm not able to figure out how? I did create the consumer group, but how do i connect the newly created consumer group to different subscriptions in my azure aaccount and get the events streamed to the eventhub that i just created?
[Note : I have followed - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create to create an evenhub on azure]
Just in case if i need to clarify on what subscription I'm talking about, below is the screenshot
How do i achieve this?
Thank you in advance
Maney
So i figured out a way to solve my problem (stated above). After going through Microsoft documentation and some trial and error methods, Here's how i solved it;
I have SUBSCRIPTION-1 and SUBSCRIPTION-2. I have created an eventhub in SUBSCRIPTION-2.
I go to SUBSCRIPTION-1 one and create a Resource-Group. After creating an Resource-Group, I create a EVENT-GRID. Within the eventgrid, I create a EVENT-SUBSCRIPTION that givens an option to point it to an endpoint. I chose the endpoint and selected the eventhub that was created in SUBSCRIPTION-1.
Now, i able to stream all the events from SUBSCRIPTION-1 to SUBSCRIPTION-2.
-Maney
I'm new to azure service bus and masstransit. I'm looking for a solution to a specific situation.
I have a azure service bus topic with multiple subscribers. Subscriber will receive message based on filters. I've created the topic and subscriber with code below
class Program
{
static void Main(string[] args)
{
string connectionString = "Endpoint connection string";
// the names of topics and subscriptions we'll be working with
const string topicName = "MyTestTopic";
const string allMessagesSubName = "AllMessages";
const string filteredSubName1 = "Filtered1";
const string filteredSubName2 = "Filtered2";
// let's create the topic if it doesn't already exist...
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
if (!namespaceManager.TopicExists(topicName))
{
var td = new TopicDescription(topicName);
namespaceManager.CreateTopic(td.Path);
}
if (!namespaceManager.SubscriptionExists(topicName, allMessagesSubName))
{
namespaceManager.CreateSubscription(topicName, allMessagesSubName);
}
if (!namespaceManager.SubscriptionExists(topicName, filteredSubName1))
{
namespaceManager.CreateSubscription(
new SubscriptionDescription(topicName, filteredSubName1),
new Microsoft.ServiceBus.Messaging.SqlFilter("From LIKE '%Smith'"));
}
if (!namespaceManager.SubscriptionExists(topicName, filteredSubName2))
{
namespaceManager.CreateSubscription(
new SubscriptionDescription(topicName, filteredSubName2),
new Microsoft.ServiceBus.Messaging.SqlFilter("sys.Label='important'"));
}
var message1 = new BrokeredMessage("Hello World");
var message2 = new BrokeredMessage("Second message");
message2.Label = "important";
var message3 = new BrokeredMessage("Third message");
message3.Properties["From"] = "Kelly Smith";
message3.Label = "information";
var client = TopicClient.CreateFromConnectionString(connectionString, topicName);
client.Send(message1);
client.Send(message2);
client.Send(message3);
client.Close();
}
}
Here in the code we're adding Message custom properties like From.
Now I want to send such message using masstransit. In masstransit I cannot find any option of adding Message custom properties using the Publish() method. Is there any way that I can send these messages using masstransit where these filters can be used?
NB: I've read the answer of this question But the anwer here tells us to filter the messages in subscriber side. What I want is that this filtering will occur before reaching the subscriber.
When using Azure Service Bus with MassTransit, you can add subscription endpoints in additional to regular endpoints. When configuring a subscription endpoint, you should be able to specify rules and/or filters as part of the subscription. Which is exactly what you're doing above, so that is handled.
The other part, adding properties to the message, can be done by adding text headers to the SendContext. Those headers are copied to the message Properties collection, which I believe can be used to filter messages using a "SQL" filter (which is configured on the subscription endpoint, or the topic subscription on a receive endpoint).
I have a listen access on Topic/Subscription for Azure Service Bus. Is there a way to verify if a subscription has a message and how many messages are there. I don't have manage connection string and I think I cannot run Service Bus Explorer. I don't intent to read the message out of the subscription.
The count of the messages in a Topic Subscription can be retrieved only using Manage Connection string. With the Listen Connection string, you cannot read the properties or count of messages.
But, you can peek the messages in the Subscription using Listen Connection string. Peeking the messages will not lock or remove the messages from the Subscription. You can peek the messages as many times as you need.
Thus, with the help of Listen Connection string, you can find the number of messages in a Subscription by peeking the messages one by one inside a while loop and add a counter inside the loop. The counter value is the number of messages inside the Subscription.
I don't intent to read the message out of the subscription.
The short answer is no. Manage access is required to get the topic or subscription message information.
If you want to get the subscription message count, you could have a try following 2 ways:
1.You have manage access, you could use the following demo code to do that.
var topicName = "topic1";
var subscriptionName = "subscription";
var address = "sb://xxx.servicebus.windows.net/"; //base address of namespace you are connecting to.
MessagingFactorySettings MsgFactorySettings = new MessagingFactorySettings
{
TokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider("SharedAccessKeyName", "xxxxxx"),
};
MessagingFactory msgFactory = MessagingFactory.Create(address, MsgFactorySettings);
NamespaceManager manager = new NamespaceManager(msgFactory.Address,msgFactory.GetSettings().TokenProvider);
var subscriptioncDescription = manager.GetSubscription(topicName, subscriptionName);
var count = subscriptioncDescription.MessageCountDetails.ActiveMessageCount;
2.Use Azure Microsoft.Azure.Management.ServiceBus.Fluent sdk. You need to registry Azure AD application and assign corresponding role it. For more information please refer to another SO thread.
var subscriptionId = "Azure subscriptionId";
var resourceGroupName = "rgName";
var nameSpace = "tomtestsb";
var topicName = "topic1";
var subscriptionName = "subscription";
var credentials = SdkContext.AzureCredentialsFactory.FromFile(#"D:\Tom\Documents\azureCred.txt");
var client = new ServiceBusManagementClient(credentials);
client.SubscriptionId = subscriptionId;
var topic = client.Topics.GetAsync(resourceGroupName, nameSpace, topicName).Result;
var topicMessagecount = topic.CountDetails.ActiveMessageCount;
var subscription = client.Subscriptions.GetAsync(resourceGroupName, nameSpace, topicName, subscriptionName).Result;
var subscriptionMessagecount = subscription.CountDetails.ActiveMessageCount;
I need to create Azure CDN Endpoint for Azure Container. I am using below code to do so.
Endpoint endpoint = new Endpoint() {
IsHttpAllowed = true,
IsHttpsAllowed = true,
Location = this.config.ResourceLocation,
Origins = new List<DeepCreatedOrigin> { new DeepCreatedOrigin(containerName, string.Format(STORAGE_URL, storageAccountName)) },
OriginPath = "/" + containerName,
};
await this.cdnManagementClient.Endpoints.CreateAsync(this.config.ResourceGroupName, storageAccountName, containerName, endpoint);
All the information I provide is correct and Endpoint is getting created successfully. But when I try to access any blob inside it. It is giving an InvalidUrl error.
However the weird thing is If I create the same endpoint using same values through portal, I am able to access and download blobs.
Anyone please let me know what am I doing wrong in my code? Do I need to pass any extra parameters?
As far as I know, if you want to create a storage CDN in code, you need set the OriginHostHeader value as your storage account URL.
More details, you could refer to below codes:
// Create CDN client
CdnManagementClient cdn = new CdnManagementClient(new TokenCredentials(token))
{ SubscriptionId = subscriptionId };
//ListProfilesAndEndpoints(cdn);
Endpoint e1 = new Endpoint()
{
// OptimizationType = "storage",
Origins = new List<DeepCreatedOrigin>() { new DeepCreatedOrigin("{yourstoragename}-blob-core-windows-net", "{yourstoragename}.blob.core.windows.net") },
OriginHostHeader = "{yourstoragename}.blob.core.windows.net",
IsHttpAllowed = true,
IsHttpsAllowed = true,
OriginPath=#"/foo2",
Location = "EastAsia"
};
cdn.Endpoints.Create(resourcegroup, profilename, enpointname, e1);
Besides, I suggest you could generate SAS token to directly access the blob file by URL.