I have this in-process function:
[CosmosDBTrigger(
databaseName: <HERE_EDIT>,
collectionName: <HERE_EDIT>,
ConnectionStringSetting = "...",
LeaseCollectionName = "...",
LeaseCollectionPrefix = <HERE_EDIT>,
StartFromBeginning = true
)]IReadOnlyList<Document> input,
[EventHub(<HERE>, Connection = <HERE>)] IAsyncCollector<EventData> collector,
ILogger log)
(...)
And I want to use values from the app settings or local.settings.json so that I can have different values depending on the deployment slot ( to substitute where it says <HERE> ).
Is that even possible?
Thanks
--- EDIT ---
As pointed by AdaTheDev I can just use the Connection prop including the name of the event hub.
But if I wanted to use in the <HERE_EDIT> to change the collection in production and the lease prefix is there a common way, that I could also apply to other attributes?
Yes. For Connection, it can be the name of an appsetting. And if that connection string includes the event hub name, that will override the value given explicitly as the EventHubName argument in the attribute.
Refence: Event Hubs Output binding
Related
I'm using Pulumi 1.16 with dotnet/C# and the AzureNative stack. I try to create an EventGridTopic. To access the created resource's properties later I pull some output values.
Example code:
var topic = new Topic("eventgrid-topic-status", new TopicArgs
{
TopicName = "egt-status-dev",
ResourceGroupName = "rg-testapp-dev",
Location = "westeurope"
});
var endPointOutput = topic.Endpoint;
var endPointAccessKey = ""; // missing output property
The resource is being created. I found no way to get the access key properties:
PrimaryAccessKey
SecondaryAccessKey
In the former (elder) Azure stack the properties exist. But in Azure Native stack not. Is that on purpose, just work in progress, has been forgotten or is there some other way to retrieve these properties on this object?
This is output on Azure (old stack):
This is Azure Native, clearly the keys are missing:
I doubt that this happens accidentally and would like to understand what to do.
Azure API (and therefore Azure Native resources) return no sensitive information in their outputs automatically to minimize security risks. You have to make an explicit call to retrieve those.
In this case, you likely need to invoke the function listTopicSharedAccessKeys.
You will want to call the function from within an Apply to make sure that it's triggered only after the topic is created (e.g., not during preview):
var keys = topic.Name.Apply(topicName => ListTopicSharedAccessKeys.InvokeAsync(
new ListTopicSharedAccessKeysArgs
{
ResourceGroupName = "rg-testapp-dev",
TopicName = topicName
}));
If you don't want to hardcode the resource group name:
let keys = pulumi.all([rg.name, topic.name]).apply(arr =>
azn.eventgrid.listTopicSharedAccessKeys(
{
resourceGroupName: arr[0],
topicName: arr[1]
}
)
);
keys.apply(x => pulumi.log.info(x.key1 ?? ""));
My EventHub has millions of messages ingestion every day. I'm processing those messages from Azure Function and printing offset and squence number value in logs.
public static async Task Run([EventHubTrigger("%EventHub%", Connection = "EventHubConnection", ConsumerGroup = "%EventHubConsumerGroup%")]EventData eventMessage,
[Inject]ITsfService tsfService, [Inject]ILog log)
{
log.Info($"PartitionKey {eventMessage.PartitionKey}, Offset {eventMessage.Offset} and SequenceNumber {eventMessage.SequenceNumber}");
}
Log output
PartitionKey , Offset 78048157161248 and SequenceNumber 442995283
Questions
PartitionKey value blank? I have 2 partitions in that EventHub
Is there any way to check backlogs? Some point of time I want to get how many messages my function need to process.
Yes, you can include the PartitionContext object as part of the signature, which will give you some additional information,
public static async Task Run([EventHubTrigger("HubName",
Connection = "EventHubConnectionStringSettingName",
ConsumerGroup = "Consumer-Group-If-Applicable")] EventData[] messageBatch, PartitionContext partitionContext, ILogger log)
Edit your host.json and set enableReceiverRuntimeMetric to true, e.g.
"version": "2.0",
"extensions": {
"eventHubs": {
"batchCheckpointFrequency": 100,
"eventProcessorOptions": {
"maxBatchSize": 256,
"prefetchCount": 512,
"enableReceiverRuntimeMetric": true
}
}
}
You now get access to RuntimeInformation on the PartitionContext, which has some information about the LastSequenceNumber, and your current message has it's own sequence number, so you could use the difference between these to calculate a metric, e.g something like,
public class EventStreamBacklogTracing
{
private static readonly Metric PartitionSequenceMetric =
InsightsClient.Instance.GetMetric("PartitionSequenceDifference", "PartitionId", "ConsumerGroupName", "EventHubPath");
public static void LogSequenceDifference(EventData message, PartitionContext context)
{
var messageSequence = message.SystemProperties.SequenceNumber;
var lastEnqueuedSequence = context.RuntimeInformation.LastSequenceNumber;
var sequenceDifference = lastEnqueuedSequence - messageSequence;
PartitionSequenceMetric.TrackValue(sequenceDifference, context.PartitionId, context.ConsumerGroupName,
context.EventHubPath);
}
}
I wrote an article on medium that goes into a bit more detail and show how you might consume the data in grafana,
https://medium.com/#dylanm_asos/azure-functions-event-hub-processing-8a3f39d2cd0f
PartitionKey value blank? I have 2 partitions in that EventHub
The partition key is not the same as the partition ids. When you publish an event to Event Hubs, you can set the partition key. If that partition key is not set, then it will be null when you go to consume it.
Partition key is for events where you don't care what partition it ends up in, just that you want events with the same key to end up in the same partition.
An example would be if you had hundreds of IoT devices transmitting telemetry data. You don't care what partition these IoT devices publish their data to, as long as it always ends up in the same partition. You may set the partition key to the serial number of the IoT device.
When that device publishes its event data with that key, the Event Hubs service will calculate a hash for that partition key, map it to a specific Event Hub partition, and will route any events with that key to the same partition.
The documentation from "Event Hubs Features: Publishing an Event" depicts it pretty well.
An exception of type 'System.FormatException' occurred in Microsoft.WindowsAzure.Storage.dll but was not handled in user code
Additional information: Settings must be of the form "name=value".
while creating eventProcessorHost in EventHub
var eventProcessorHost = new EventProcessorHost(
EhEntityPath,
EventHubConsumerGroup.DefaultGroupName,
EhConnectionString,
StorageConnectionString,
StorageContainerName);
Sample values
string EhConnectionString = "Endpoint=sb://namespacename-ns.servicebus.windows.net/;SharedAccessKeyName=receivepolicy;SharedAccessKey=v7IHIH+jB3+H2UMxEOr9kHYfhwj1Q=;EntityPath=sampleeventhub";
string EhEntityPath = "sampleeventhub";
string StorageContainerName = "containername"; //I have created in blob and type as container
string StorageAccountName = "storegenameinazure";
string StorageAccountKey = "GHasmRRJgI5s123ziDlfOKQ7IBrO23EvHpk++TV0L2hU2erdI7PyY+gtvUop67lIU0+zQsM09sQ==";
static readonly string StorageConnectionString = string.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", StorageAccountName, StorageAccountKey);
According to your description, I leverage Microsoft.Azure.ServiceBus.EventProcessorHost version (2.2.10) to test this issue.
An exception of type 'System.FormatException' occurred in Microsoft.WindowsAzure.Storage.dll but was not handled in user code. Additional information: Settings must be of the form "name=value".
Based on your code, I assumed that you have not built your EventProcessorHost constructor correctly, the constructor you used looks like this:
At this point, the parameter StorageContainerName you passed would be treated as storageConnectionString parameter, then you got the above error.
In summary, please build your EventProcessorHost with the correct constructor. Here are some tutorials you could refer to them (event-processor-host-best-practices and azure-event-hubs-processing).
I was trying to keep the history of data (at least one step back) of DocumentDB.
For example, if I have a property called Name in document with value "Pieter". Now I am changing that to "Sam", I have to maintain the history , it was "Pieter" previously.
As of now I am thinking of a pre-trigger. Any other solutions ?
Cosmos DB (formerly DocumentDB) now offers change tracking via Change Feed. With Change Feed, you can listen for changes on a particular collection, ordered by modification within a partition.
Change feed is accessible via:
Azure Functions
DocumentDB (SQL) SDK
Change Feed Processor Library
For example, here's a snippet from the Change Feed documentation, on reading from the Change Feed, for a given partition (full code example in the doc here):
IDocumentQuery<Document> query = client.CreateDocumentChangeFeedQuery(
collectionUri,
new ChangeFeedOptions
{
PartitionKeyRangeId = pkRange.Id,
StartFromBeginning = true,
RequestContinuation = continuation,
MaxItemCount = -1,
// Set reading time: only show change feed results modified since StartTime
StartTime = DateTime.Now - TimeSpan.FromSeconds(30)
});
while (query.HasMoreResults)
{
FeedResponse<dynamic> readChangesResponse = query.ExecuteNextAsync<dynamic>().Result;
foreach (dynamic changedDocument in readChangesResponse)
{
Console.WriteLine("document: {0}", changedDocument);
}
checkpoints[pkRange.Id] = readChangesResponse.ResponseContinuation;
}
If you're trying to make an audit log I'd suggest looking into Event Sourcing.Building your domain from events ensures a correct log. See https://msdn.microsoft.com/en-us/library/dn589792.aspx and http://www.martinfowler.com/eaaDev/EventSourcing.html
I'm attempting to modify the Azure-based Video Store sample app so that the front-end Ecommerce site can scale out.
Specifically, I want all instances of the web site to be notified of events like OrderPlaced so that no matter which web server the client web app happens to be connected to via SignalR, it will correctly receive the notification and update the UI.
Below is my current configuration in the Global.asax:
Feature.Disable<TimeoutManager>();
Configure.ScaleOut(s => s.UseUniqueBrokerQueuePerMachine());
startableBus = Configure.With()
.DefaultBuilder()
.TraceLogger()
.UseTransport<AzureServiceBus>()
.PurgeOnStartup(true)
.UnicastBus()
.RunHandlersUnderIncomingPrincipal(false)
.RijndaelEncryptionService()
.CreateBus();
Configure.Instance.ForInstallationOn<Windows>().Install();
bus = startableBus.Start();
And I've also configured the Azure Service Bus queues using:
class AzureServiceBusConfiguration : IProvideConfiguration<NServiceBus.Config.AzureServiceBusQueueConfig>
{
public AzureServiceBusQueueConfig GetConfiguration()
{
return new AzureServiceBusQueueConfig()
{
QueuePerInstance = true
};
}
}
I've set the web role to scale to two instances, and as expected, two queues (ecommerce and ecommerce-1) are created. I do not, however, see additional topic subscriptions being created under the videostore.sales.events topic. Instead, I see:
I would think that you would see VideoStore.ECommerce-1.OrderCancelled and VideoStore.ECommerce-1.OrderPlaced subscriptions under the Videostore.Sales.Events topic. Or is that not how subscriptions are stored when using Azure Service Bus?
What am I missing here? I get the event on one of the ecommerce instances, but never on both. Even if this isn't the correct way to scale out SignalR, my use case extends to stuff like cache invalidation.
I also find it strange that two error and audit queues are being created. Why would that happen?
UPDATE
Yves is correct. The AzureServiceBusSubscriptionNamingConvention was not applying the correct individualized name. I was able to fix this by implementing the following EndpointConfig:
namespace VideoStore.ECommerce
{
public class EndpointConfig : IConfigureThisEndpoint, IWantCustomInitialization
{
public void Init()
{
AzureServiceBusSubscriptionNamingConvention.Apply = BuildSubscriptionName;
AzureServiceBusSubscriptionNamingConvention.ApplyFullNameConvention = BuildSubscriptionName;
}
private static string BuildSubscriptionName(Type eventType)
{
var subscriptionName = eventType != null ? Configure.EndpointName + "." + eventType.Name : Configure.EndpointName;
if (subscriptionName.Length >= 50)
subscriptionName = new DeterministicGuidBuilder().Build(subscriptionName).ToString();
if (!SettingsHolder.GetOrDefault<bool>("ScaleOut.UseSingleBrokerQueue"))
subscriptionName = Individualize(subscriptionName);
return subscriptionName;
}
public static string Individualize(string queueName)
{
var parser = new ConnectionStringParser();
var individualQueueName = queueName;
if (SafeRoleEnvironment.IsAvailable)
{
var index = parser.ParseIndexFrom(SafeRoleEnvironment.CurrentRoleInstanceId);
var currentQueue = parser.ParseQueueNameFrom(queueName);
if (!currentQueue.EndsWith("-" + index.ToString(CultureInfo.InvariantCulture))) //individualize can be applied multiple times
{
individualQueueName = currentQueue
+ (index > 0 ? "-" : "")
+ (index > 0 ? index.ToString(CultureInfo.InvariantCulture) : "");
}
if (queueName.Contains("#"))
individualQueueName += "#" + parser.ParseNamespaceFrom(queueName);
}
return individualQueueName;
}
}
}
I could not, however, get NServiceBus to recognize my EndpointConfig class. Instead, I had to call it manually before starting the bus. From my Global.asax.cs:
new EndpointConfig().Init();
bus = startableBus.Start();
Once I did this, the subscription names appeared as expected:
Not sure why it's ignoring my IConfigureThisEndpoint, but this works.
This sounds like a bug, can you raise a github issue on this at https://github.com/Particular/NServiceBus.Azure
That said, I think it's better to use signalr's scaleout feature instead of using QueuePerInstance as signalr needs to replicate other information like (connection/group mappings) internally as well when running in scaleout mode.
Update:
I think I see the issue, the subscriptions should be individualised as well, which isn't the case in current naming conventions
https://github.com/Particular/NServiceBus.Azure/blob/master/src/NServiceBus.Azure.Transports.WindowsAzureServiceBus/NamingConventions/AzureServiceBusSubscriptionNamingConvention.cs
while it is in the queuenamingconventions
https://github.com/Particular/NServiceBus.Azure/blob/master/src/NServiceBus.Azure.Transports.WindowsAzureServiceBus/NamingConventions/AzureServiceBusQueueNamingConvention.cs#L27
As these conventions are public you can override them to work around the problem by changing the func in IWantCustomInitialization until I can get a fix in, just copy the current method and add the individualizer logic. The queue individualizer is internal though, so you'll have to copy that class from
https://github.com/Particular/NServiceBus.Azure/blob/master/src/NServiceBus.Azure.Transports.WindowsAzureServiceBus/Config/QueueIndividualizer.cs