Convert QueueClient.Create to MessagingFactory.CreateQueueClient - azure

Trying to convert an implementation using the .net library from using QueueClient.Create to the MessagingFactory.CreateQueueClient to be able to better control the BatchFlushInterval as well as to allowing the use of multiple factories over multiple connections to increase send throughput but running into roadblocks.
Right now we are creating QueueClients (they are maintained throughout the app) like this:
QueueClient.CreateFromConnectionString(address, queueName, ReceiveMode.PeekLock); // address is the connection string from the azure portal in the form of Endpoint=sb....
Trying to change it to creating a MessagingFactory in the class construtor that will be used to create the QueueClients:
messagingFactory = MessagingFactory.Create(address.Replace("Endpoint=",""),mfs);
// later on in another part of the class
messagingFactory.CreateQueueClient(queueName, ReceiveMode.PeekLock);
// error Endpoint not found.,
This throws the error Endpoint not found. If I don't replace the Endpoint= it won't even create the MessagingFactory. What is the proper way to handle this?
Notes:
address = Endpoint=sb://pmg-bus-mybus.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=somekey
As an aside, we have a process that is trying to push as many messages as possible to a queue and others reading it. The readers seem to easily keep up with the sender and I'm trying to maximize the send rate.

The address is the base address of namespace(sb://yournamespace.servicebus.windows.net/) you are connecting to. For more information, please refer to MessagingFactory. The following is the demo code :
var Address = "sb://yournamespace.servicebus.windows.net/"; //base address of namespace you are connecting to.
MessagingFactorySettings MsgFactorySettings = new MessagingFactorySettings
{
NetMessagingTransportSettings = new NetMessagingTransportSettings
{
BatchFlushInterval = TimeSpan.FromSeconds(2)
},
TokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider("RootManageSharedAccessKey", "balabala..."),
OperationTimeout = TimeSpan.FromSeconds(30)
}; //specify operating timeout (optional)
MessagingFactory messagingFactory = MessagingFactory.Create(Address, MsgFactorySettings);
var queue = messagingFactory.CreateQueueClient("queueName",ReceiveMode.PeekLock);
var message = queue.Receive(TimeSpan.Zero);

Related

How to use CosmosClient.CreateAndInitializeAsync() with CosmosClientBuilder.Build()

This YouTube video #27:20 talks about populating the cache with routing info to avoid latency during a cold start.
You can either try to get a document you know doesn't exist, or you can use CosmosClient.CreateAndInitializeAsync().
I already have this code set up:
private async Task<Container> CreateContainerAsync(string endpoint, string authKey)
{
var cosmosClientBuilder = new CosmosClientBuilder(
accountEndpoint: endpoint,
authKeyOrResourceToken: authKey)
.WithConnectionModeDirect(portReuseMode: PortReuseMode.PrivatePortPool, idleTcpConnectionTimeout: TimeSpan.FromHours(1))
.WithApplicationName(UserAgentSuffix)
.WithConsistencyLevel(ConsistencyLevel.Session)
.WithApplicationRegion(Regions.AustraliaEast)
.WithRequestTimeout(TimeSpan.FromSeconds(DatabaseRequestTimeoutInSeconds))
.WithThrottlingRetryOptions(TimeSpan.FromSeconds(DatabaseMaxRetryWaitTimeInSeconds), DatabaseMaxRetryAttemptsOnThrottledRequests);
var client = cosmosClientBuilder.Build();
var databaseResponse = await CreateDatabaseIfNotExistsAsync(client).ConfigureAwait(false);
var containerResponse = await CreateContainerIfNotExistsAsync(databaseResponse.Database).ConfigureAwait(false);
return containerResponse;
}
Is there any way to incorporate CosmosClient.CreateAndInitializeAsync() with it to populate the cache?
If not, is it ok to do this to populate the cache?
public class CosmosClientWrapper
{
public CosmosClientWrapper(IKeyVaultFacade keyVaultFacade)
{
var container = CreateContainerAsync(endpoint, authenticationKey).GetAwaiter().GetResult();
// Get a document that doesn't exist to populate the routing info:
container.ReadItemAsync<object>(Guid.NewGuid().ToString(), PartitionKey.None).GetAwaiter().GetResult();
}
}
The point of CreateAndInitialize or BuildAndInitialize is to pre-establish the connections required to perform Data Plane operations to the desired containers (reference https://learn.microsoft.com/azure/cosmos-db/nosql/sdk-connection-modes#routing).
If the containers do not exist, then it makes no sense to use CreateAndInitialize or BuildAndInitialize because there are no connections that can be pre-established/warmed up, because there are no target backend endpoints to connect to. That is why the container/database information is required, because the only benefit is warming up the connections to the backend machines that support that/those container/s.
Please see CosmosClientBuilder.BuildAndInitializeAsync which creates the cosmos client and initialize the provided containers. I believe this is what you are looking for.

Azure Cosmos DB client throws "HttpRequestException: attempt was made to access a socket in a way forbidden by its access permissions" underneath

I use CosmosClient from SDK Microsoft.Azure.Cosmos 3.28.0 in ASP.NET Core 3.1 in Azure Durable Function. This client is getting and sending data from/to my cosmos instance (Core (SQL)) and it works fine but I see that it constantly throws exception in following http request for metadata
GET 169.254.169.254/metadata/instance
System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions.
I use following configuration:
private static void RegisterCosmosDbClient(ContainerBuilder builder)
{
builder.Register(c => new SocketsHttpHandler()
{
PooledConnectionLifetime = TimeSpan.FromMinutes(10), // Customize this value based on desired DNS refresh timer
MaxConnectionsPerServer = 20, // Customize the maximum number of allowed connections
}).As<SocketsHttpHandler>().SingleInstance();
builder.Register(
x =>
{
var cosmosDbOptions = x.Resolve<CosmosDbOptions>();
var socketsHttpHandler = x.Resolve<SocketsHttpHandler>();
return new CosmosClient(cosmosDbOptions.ConnectionString, new CosmosClientOptions()
{
ConnectionMode = ConnectionMode.Direct,
PortReuseMode = PortReuseMode.PrivatePortPool,
IdleTcpConnectionTimeout = new TimeSpan(0, 23, 59, 59),
SerializerOptions = new CosmosSerializationOptions()
{
PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase
},
HttpClientFactory = () => new HttpClient(socketsHttpHandler, disposeHandler: false)
});
})
.AsSelf()
.SingleInstance();
}
I also tried approach with passing IHttpClientFactory from this blog but it didn't help.
It looks like there are no new sockets available in your environment therefore you are getting the socket forbidden error. Please review how to manage connection for Azure Cosmos DB clients and you should use a singleton Azure Cosmos DB client for the lifetime of your application to resolve the issue. In case if you still facing the issue leveraging the singleton object then please let me know so we can further review it.
That particular IP and path is for https://learn.microsoft.com/azure/virtual-machines/windows/instance-metadata-service?tabs=windows
The SDK is attempting to detect the Azure information. It could mean for Durable Functions, this information and endpoint is not available.
This does not affect SDK operations and should not block you from performing other actions on the CosmosClient instance.

What's the equivalent of NamespaceManager.GetQueue.MessageCount in the new NET Standard Microsoft.Azure.ServiceBus?

What's the equivalent of NamespaceManager in the new NET Standard Microsoft.Azure.ServiceBus?
I've used WindowsAzure.ServiceBus to do things like count messages in a queue ...
var namespaceManager = NamespaceManager.CreateFromConnectionString(SbConnectionString);
var count = namespaceManager.GetQueue(queueName).MessageCount;
Moving over to the new Microsoft.Azure.ServiceBus .NET Standard library, but whilst it's got classes like QueueClient and TopicClient, it's not got any NamespaceManager
How can you could do message counts in the new .NET Standard library?
To provide an update:
This is now implemented and available under the Microsoft.Azure.ServiceBus.Management; namespace.
NamespaceManager is now called ManagementClient and has (roughly) the same endpoints available.
Here's the class itself as part of the pull request to merge it into the main repository.
You can't.
The new API doesn't support reading message counts. You'd have to use Azure Monitor API to get them.
Read why that's that case and how to work with Azure Monitor in Reading Azure Service Bus Metrics.
You CAN read the queue message count:
var managementClient = new ManagementClient(connectionString);
(await managementClient.GetQueueRuntimeInfoAsync("name")).MessageCount;
using Azure.Messaging.ServiceBus.Administration;
private readonly ServiceBusAdministrationClient _serviceBusAdministrationClient;
_serviceBusAdministrationClient = new ServiceBusAdministrationClient(connectionString);
Response<QueueRuntimeProperties> queueRuntimePropertiesAsync = await _serviceBusAdministrationClient.GetQueueRuntimePropertiesAsync(name);
ActiveMessageCount = queueRuntimeProperties.ActiveMessageCount,
DeadLetterMessageCount = queueRuntimeProperties.DeadLetterMessageCount,
ScheduledMessageCount = queueRuntimeProperties.ScheduledMessageCount,
TotalMessageCount = queueRuntimeProperties.TotalMessageCount,

How to use Pinpoint send messages to specific devices using the Java AWS SDK for Pinpoint

I have been able to get my mobile Android app to receive messages generated from the Pinpoint Campaign console (https://console.aws.amazon.com/pinpoint/home) to a specific device by targeting the segment to a custom attribute that only that device has.
Pinpoint Campaign config
Mobile push channel
Standard campaign
Segment defined using custom attributes, holdout 0%
Silent notification
Custom JSON
Launch immediate
Now I would like to implement this feature in my Java app using the SDK APIs and target the device's Pinpoint endpoint.
GetEndpointRequest getEndpointRequest = new GetEndpointRequest()
.withApplicationId(appId)
.withEndpointId(endpointId);
GetEndpointResult endpointResult = getAmazonPinpointClient().getEndpoint(getEndpointRequest);
DirectMessageConfiguration directMessageConfiguration =
new DirectMessageConfiguration().withGCMMessage(new GCMMessage().withBody(body).withSilentPush(true).withAction(Action.OPEN_APP));
AddressConfiguration addressConfiguration = new AddressConfiguration().withChannelType(ChannelType.GCM);
MessageRequest messageRequest = new MessageRequest().withMessageConfiguration(directMessageConfiguration)
.addAddressesEntry(endpointResponse.getAddress(), addressConfiguration);
SendMessagesRequest sendMessagesRequest = new SendMessagesRequest()
.withApplicationId(appId)
.withMessageRequest(messageRequest);
The "body" is the same JSON I put in the Pinpoint Campaign console. When I run this, I get back a DeliveryStatus of SUCCESSFUL but the device never receives the message.
{ApplicationId: MY_APP_ID,Result: {clrVUcv-AwA:APA91bHGXkxpDJiw5kOMROA2XTJXuKreMklq9jemHO_KGYTIw6w84Fw9zLv9waMgLgha61IR-kZxgmrnFu-OGp8l6WFgp4Wolh4oOvZwMobGYNgzivv3bGIK83t-e4hiLx1TTaEIeRdQ={DeliveryStatus: SUCCESSFUL,StatusCode: 200,StatusMessage: {"multicast_id":4803589342422496921,"success":1,"failure":0,"canonical_ids":0,"results":[{"message_id":"0:1515105369948916%c551fa42f9fd7ecd"}]},}}}
I have also tried this via the AWS CLI:
aws pinpoint send-messages --application-id MY_APP_ID --message-request "{\"Addresses\":{\"clrVUcv-AwA:APA91bHGXkxpDJiw5kOMROA2XTJXuKreMklq9jemHO_KGYTIw6w84Fw9zLv9waMgLgha61IR-kZxgmrnFu-OGp8l6WFgp4Wolh4oOvZwMobGYNgzivv3bGIK83t-e4hiLx1TTaEIeRdQ\":{\"ChannelType\":\"GCM\"}},\"MessageConfiguration\":{\"GCMMessage\":{\"Body\":\"{\\\"message\\\":\\\"stuff\\\"}\",\"SilentPush\":true}}}"
with a similar result (get 200 status code and DeliveryStatus of SUCCESSFUL but the app never receives). I tried using the "Direct" message in the AWS Pinpoint console but they do not seem to support the same format (forces Action and Title/Message instead of silent push message with custom JSON).
Am I getting the endpoint incorrectly? How do I translate the above campaign into a message? I see there is a sendUserMessages() API call as well but that doesn't seem to be right one (I couldn't find where to specify the specific user endpoint)?
The client receives the campaign via the registered Service:
public class PushListenerService extends GcmListenerService {
#Override
public void onMessageReceived(final String from, final Bundle data) {
AWSMobileClient.initializeMobileClientIfNecessary(this.getApplicationContext());
final NotificationClient notificationClient = AWSMobileClient.defaultMobileClient()
.getPinpointManager().getNotificationClient();
NotificationClient.CampaignPushResult pushResult =
notificationClient.handleGCMCampaignPush(from, data, this.getClass());
Log.e(LOG_TAG, " onMessageReceived - got messages" + data);
Do GCM direct messages get sent through the same campaign method or do I have to register a different service to process these?
Found the solution based on the AWS CLI command I was able to run. Should have been using the "Data" element and not the "Body" and need to enable "SilentPush".
EndpointResponse endpointResponse = getPinpointEndpointResponse(appId, pinpointEndpointId);
Map<String, String> data = new HashMap<>();
// construct data here, currently only supports Map<String, String>
// why not HashMap<String, Object> so it can support full JSON????
DirectMessageConfiguration directMessageConfiguration =
new DirectMessageConfiguration().withGCMMessage(new GCMMessage().withData(data).withSilentPush(true));
AddressConfiguration addressConfiguration = new AddressConfiguration().withChannelType(ChannelType.GCM);
MessageRequest messageRequest = new MessageRequest().withMessageConfiguration(directMessageConfiguration)
.addAddressesEntry(endpointResponse.getAddress(), addressConfiguration);
SendMessagesRequest sendMessagesRequest = new SendMessagesRequest()
.withApplicationId(appId)
.withMessageRequest(messageRequest);

Same Azure topic is processed multiple times

We have a job hosted in an azure website, the job reads entries from a topic subscription. Everything works fine when we only have one instance to host the website. Once we scale out to more than one instance we observe the message is processed as many times as instances we have. Each instance points to the same subscription. From what we read, once the item is read, it won't be available for any other process. The duplicated processing is happening inside the same instance, meaning that if we have two instances, the item is processed twice in one of the instances, it is not splitted.
What can be possible be wrong in the way we are doing things?
This is how we proceed to configure the connection to the queue, if the subscription does not exists, it is created:
var serviceBusConfig = new ServiceBusConfiguration
{
ConnectionString = transactionsBusConnectionString
};
config.UseServiceBus(serviceBusConfig);
var allRule1 = new RuleDescription
{
Name = "All",
Filter = new TrueFilter()
};
SetupSubscription(transactionsBusConnectionString,"topic1", "subscription1", allRule1);
private static void SetupSubscription(string busConnectionString, string topicNameKey, string subscriptionNameKey, RuleDescription newRule)
{
var namespaceManager =
NamespaceManager.CreateFromConnectionString(busConnectionString);
var topicName = ConfigurationManager.AppSettings[topicNameKey];
var subscriptionName = ConfigurationManager.AppSettings[subscriptionNameKey];
if (!namespaceManager.SubscriptionExists(topicName, subscriptionName))
{
namespaceManager.CreateSubscription(topicName, subscriptionName);
}
var subscriptionClient = SubscriptionClient.CreateFromConnectionString(busConnectionString, topicName, subscriptionName);
var rules = namespaceManager.GetRules(topicName, subscriptionName);
foreach (var rule in rules)
{
subscriptionClient.RemoveRule(rule.Name);
}
subscriptionClient.AddRule(newRule);
rules = namespaceManager.GetRules(topicName, subscriptionName);
rules.ToString();
}
Example of the code that process the topic item:
public void SendInAppNotification(
[ServiceBusTrigger("%eventsTopicName%", "%SubsInAppNotifications%"), ServiceBusAccount("OutputServiceBus")] Notification message)
{
this.valueCalculator.AddInAppNotification(message);
}
This method is inside a Function static class, I'm using azure web job sdk.
Whenever the azure web site is scaled to more than one instance, all the instances share the same configuration.
It sounds like you're creating a new subscription each time your new instance runs, rather than hooking into an existing one. Topics are designed to allow multiple subscribers to attach in that way as well - usually though each subscriber has a different purpose, so they each see a copy of the message.
I cant verify this from your code snippet but that's my guess - are the config files identical? You should add some trace output to see if your processes are calling CreateSubscription() each time they run.
I think I can access the message id, I'm using azure web job sdk but I think I can find a way to get it. Let me check it and will let you know.

Resources