I’ve got a WinRT app that I’m using the Windows Azure Toolkit for Windows 8 with. I’ve got a setup where I’d like clients subscribed to ignore messages posted to a ServiceBus Topic if they’re the originator or if the message is older than when their subscription started.
In the Properties of my BrokeredMessage, I’ve added 2 items to cover these scenarios:
message.Properties["Timestamp"] = DateTime.UtcNow.ToFileTime();
message.Properties["OriginatorId"] = clientId.ToString();
clientId is a Guid.
The subscriber side looks like this:
// ti is a class that contains a Topic, Subscription and a bool as a cancel flag.
string FilterName = "NotMineNewOnly";
// Find or create the topic.
if (await Topic.ExistsAsync(DocumentId.ToString(), TokenProvider))
{
ti.Topic = await Topic.GetAsync(DocumentId.ToString(), TokenProvider);
}
else
{
ti.Topic = await Topic.CreateAsync(DocumentId.ToString(), TokenProvider);
}
// Find or create this client's subscription to the board.
if (await ti.Topic.Subscriptions.ExistsAsync(ClientSettings.Id.ToString()))
{
ti.Subscription = await ti.Topic.Subscriptions.GetAsync(ClientSettings.Id.ToString());
}
else
{
ti.Subscription = await ti.Topic.Subscriptions.AddAsync(ClientSettings.Id.ToString());
}
// Find or create the subscription filter.
if (!await ti.Subscription.Rules.ExistsAsync(FilterName))
{
// Want to ignore messages generated by this client and ignore any that are older than Timestamp.
await ti.Subscription.Rules.AddAsync(FilterName, sqlFilterExpression: string.Format("(OriginatorId != '{0}') AND (Timestamp > {1})", ClientSettings.Id, DateTime.UtcNow.ToFileTime()));
}
ti.CancelFlag = false;
Topics[boardId] = ti;
while (!ti.CancelFlag)
{
BrokeredMessage message = await ti.Subscription.ReceiveAndDeleteAsync(TimeSpan.FromSeconds(30));
if (!ti.CancelFlag && message != null)
{
// Everything gets here! :(
}
I get back everything – so I’m not sure what I’m doing wrong. What’s the easiest way to troubleshoot problems with subscription filters?
When you create a Subscription then by default you get a "MatchAll" filter. In the code above you are just adding your filter so it is applied in addition to the "MatchAll" filter and thus all messages are recieved. Just delete the $Default filter once the Subscription is created and that should resolve the issue.
Best way to troubleshoot is using the Service Bus Explorer from Paolo Salvatori
http://code.msdn.microsoft.com/windowsazure/Service-Bus-Explorer-f2abca5a
He has done a good few blog posts on it e.g. http://windowsazurecat.com/2011/07/exploring-topics-and-queues-by-building-a-service-bus-explorer-toolpart-1/
Windows Azure SDK 1.7 does have built in capability but the Service Bus Explorer Standalone version is still better, see comparison here.
http://soa-thoughts.blogspot.com.au/2012/06/visual-studio-service-bus-explorer.html
HTH your debugging...
Related
I want to stream the events from different subscription to a single eventhub on azure.
At present I have configured eventhub to a single subscription and events are being streamed. I have a java client which consumes these events and stores it on my persistence layer. My java client looks like..
private void processUsingProcessorClient(){
List<Disposable> subscriptions = null;
try {
EventHubConsumerAsyncClient eventHubConsumerAsyncClient = new EventHubClientBuilder()
.consumerGroup(EventHubClientBuilder.DEFAULT_CONSUMER_GROUP_NAME)
.connectionString(CONNECTION_STRING, EVENT_HUB_NAME)
.credential("*******.servicebus.windows.net","maney-event-hub",createClientSecretCredential())
.buildAsyncConsumerClient();
ReceiveOptions receiveOptions = new ReceiveOptions().setTrackLastEnqueuedEventProperties(true);
List<String> block = eventHubConsumerAsyncClient.getPartitionIds().collectList().block();
Iterator<String> iterator = block.stream().iterator();
String partitionID = null;
subscriptions = new ArrayList<>(block.size());
while(iterator.hasNext()){
partitionID = iterator.next();
Disposable subscription = eventHubConsumerAsyncClient.receiveFromPartition(
partitionID,
EventPosition.fromOffset(0),receiveOptions).subscribe(PARTITION_PROCESSOR,ERROR_HANDLER);
subscriptions.add(subscription);
}
System.in.read();
}catch (Exception ex){
ex.printStackTrace();
} finally {
if(subscriptions != null){
subscriptions.forEach( subscrip -> {
subscrip.dispose();
});
}
}
}
private final Consumer<PartitionEvent> PARTITION_PROCESSOR = partitionEvent -> {
EventData event = partitionEvent.getData();
PartitionContext partitionContext = partitionEvent.getPartitionContext();
String contents = new String(event.getBody(), UTF_8);
LastEnqueuedEventProperties properties = partitionEvent.getLastEnqueuedEventProperties();
System.out.printf("Information received at %s. Last enqueued sequence number: %s%n",properties.getRetrievalTime(), properties.getSequenceNumber());
System.out.printf("Partition[%s] with Offset-[%s] and Sequence Number[%s] has contents: '%s'%n",
partitionContext.getPartitionId(),
event.getOffset(),
event.getSequenceNumber(),
contents);
};
private final Consumer<Throwable> ERROR_HANDLER = errorContext -> {
System.out.printf("Error occurred in partition processor");
errorContext.printStackTrace();
};
public ClientSecretCredential createClientSecretCredential() {
ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
.clientId("****************")
.clientSecret("******************")
.tenantId("**********************")
.build();
return clientSecretCredential;
}
I'm able to read all the events from a single subscription. However I need to do data analytics on these events from different subscriptions too. How do I configre Azure Eventhub to listen to multiple subscriptions ?
I read on Stackoverflow suggestions about creating consumer groups to solve this issue, however I'm not able to figure out how? I did create the consumer group, but how do i connect the newly created consumer group to different subscriptions in my azure aaccount and get the events streamed to the eventhub that i just created?
[Note : I have followed - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create to create an evenhub on azure]
Just in case if i need to clarify on what subscription I'm talking about, below is the screenshot
How do i achieve this?
Thank you in advance
Maney
So i figured out a way to solve my problem (stated above). After going through Microsoft documentation and some trial and error methods, Here's how i solved it;
I have SUBSCRIPTION-1 and SUBSCRIPTION-2. I have created an eventhub in SUBSCRIPTION-2.
I go to SUBSCRIPTION-1 one and create a Resource-Group. After creating an Resource-Group, I create a EVENT-GRID. Within the eventgrid, I create a EVENT-SUBSCRIPTION that givens an option to point it to an endpoint. I chose the endpoint and selected the eventhub that was created in SUBSCRIPTION-1.
Now, i able to stream all the events from SUBSCRIPTION-1 to SUBSCRIPTION-2.
-Maney
I've developed a chatbot that communicates with SharePoint on Premise,
When I run the chatbot in Emulator its work.
But When I run at Web that hosted outside of SharePoint, it does not work.
Herewith my screenshot of Error On Azure, From the result of Error is starting from XMLReader and SyndicationFeed
Success in Local Emulator
Herewith my Souce Code.
private async Task ProcessRSSAsync(ITurnContext<IMessageActivity> turnContext, LuisResult luisResult, string intent, CancellationToken cancellationToken)
{
var questionluis = turnContext.Activity.Text;
await turnContext.SendActivityAsync("intent recognize" + intent);
var intentresut = intent;
await turnContext.SendActivityAsync("Get LUIS Entity");
await turnContext.SendActivityAsync(string.Join("\t", luisResult.Entities.Select((entityObj) => entityObj.Entity)));
var entityfound = string.Join("\t", luisResult.Entities.Select((entityObj) => entityObj.Entity));
string spxurl = #"https://intra.aspac.com/sites/sg/daw/_layouts/15/srchrss.aspx?k=*%20ListId:7BC0F2C3-6366-48B8-B88A-8738BE1F9C31";
await turnContext.SendActivityAsync("Intent: " + intent.ToString() + " Entity: " + entityfound.ToString());
////---------------------------------------------------------------------------------------------
//22112019
try
{
//#ES09122019
var credentials = new NetworkCredential("email#example.com", "Pa$$w0rd", "sg.kworld.com");
var handler = new HttpClientHandler { Credentials = credentials, UseDefaultCredentials = false };
var client = new HttpClient(handler);
client.BaseAddress = new Uri("https://intra.aspac.com/sites/sg/daw/");
HttpResponseMessage resp = client.GetAsync("_layouts/15/srchrss.aspx?k=" + entityfound + "*%20ListId:7BC0F2C3-6366-48B8-B88A-8738BE1F9C31").Result;
string respString = resp.Content.ReadAsStringAsync().Result;
if (resp.StatusCode == HttpStatusCode.OK)
{
await turnContext.SendActivityAsync("Connected");
//Success 06122019 .
try
{
string spurl = #"https://intra.aspac.com/sites/sg/daw/_layouts/15/srchrss.aspx?k=*%20ListId:7BC0F2C3-6366-48B8-B88A-8738BE1F9C31";
XmlSecureResolver resolver = new XmlSecureResolver(new XmlUrlResolver(), spurl);
resolver.Credentials = new NetworkCredential("email#example.com.sg", "Pa$$w0rd", "sg.kworld.com");
XmlReaderSettings settings = new XmlReaderSettings();
settings.DtdProcessing = DtdProcessing.Parse;
settings.ValidationType = ValidationType.DTD;
settings.XmlResolver = resolver;
XmlReader reader = XmlReader.Create(spurl, settings);
SyndicationFeed feed = SyndicationFeed.Load(reader);
reader.Close();
var attachments = new List<Attachment>();
foreach (SyndicationItem item in feed.Items)
{
//Get Title,Description,URL
String title = item.Title.Text;
String description = item.Summary.Text;
String link = item.Links.FirstOrDefault().Uri.ToString();
//Hero Card
var heroCard = new HeroCard(
title: item.Title.Text,
// subtitle: description,
buttons: new CardAction[]
{
new CardAction(ActionTypes.OpenUrl,"Learn More",value:link)
}
).ToAttachment();
attachments.Add(heroCard);
}
var reply = MessageFactory.Carousel(attachments);
await turnContext.SendActivityAsync(reply);
await ProcessCosmoDBStorageLUISAsync(turnContext, questionluis, intent, entityfound, respString, cancellationToken);
}
catch (Exception ex)
{
await turnContext.SendActivityAsync(ex.ToString());
}
}
}
catch (Exception ex)
{
await turnContext.SendActivityAsync("Sorry,Currently Server Under Maintenace");
await turnContext.SendActivityAsync(ex.ToString());
}
}
any solution for this and suggestion?
ok, I think I finally understand this better, so hopefully can put a useful reply together. Would be much easier if we had a shared whiteboard :-)
Basically, in terms of hosting a bot on the Microsoft Bot Framework Services, you need to have a registration in Azure. However, there are two different options, and both are VERY different in terms of hosting. When you "create" the resource in Azure, and search for "Bot", you'll see two options - "Web App Bot" and "Bot Channels Registration":
"Bot Channels Registration" means JUST registering your bot in Azure, but HOSTING it elsewhere.
"Web App Bot" - INCLUDES the "Bot Channels Registration" but ALSO adds hosting using an Azure Web Application (so it's a Bot registration PLUS hosting)
From the screenshot you posted, I can see you've selected (2) above, and so your bot is running inside Azure, and therefore can't connect to your on premises resource (SharePoint).
As a result, I'd suggest one of two options:
Create an Azure Application Proxy - this is basically a small gateway so that your bot HOSTED in Azure can securely talk to your on-premises SharePoint. There is in fact a specific use case for SharePoint in particular.
Delete and re-create your Azure Bot entry to instead be just a "Bot Channels Registration", and then in the "Settings" screen you can call a bot hosted at any "https" endpoint. You can then have your bot run on the local network, but it will need a live "https" address (not -that- hard to do, but you have to involve your IT team to get a live web address, like "whatever.aspac.com", and you'll need an SSL/TLS certificate so that it can run httpS instead of just http.
Which option you choose might depend on the skills and resources on your team, as well as in the organisation. For instance, the company might have Azure Application Proxy configured already, in which case that saves a lot of work. It might have a wildcard certificate, which would make option (2) easier, etc.
Either way, I hope that helped, but feel free to ask more if anything is still unclear.
i had a similar problem using a on premise database. as you are deploying your bot externally, the bot needs resources that are available on the internet, and not contained internally. It will work fine using the bot emulator because it has access to what your machine has.
Saying that, azure has developed some actions which you can use to help this problem. If you look at application proxys, that may be able to help you out.
i think thats what you mean... anyway!
I am following the example in the link https://github.com/microsoftgraph/msgraph-training-changenotifications and creating and updating the notification subscription using a timer class.
[HttpGet]
public ActionResult<string> Get()
{
var graphServiceClient = GetGraphClient();
var sub = new Microsoft.Graph.Subscription();
sub.ChangeType = "updated";
sub.NotificationUrl = config.Ngrok + "/api/notifications";
sub.Resource = "/users";
sub.ExpirationDateTime = DateTime.UtcNow.AddMinutes(5);
sub.ClientState = "SecretClientState";
var newSubscription = graphServiceClient
.Subscriptions
.Request()
.AddAsync(sub).Result;
Subscriptions[newSubscription.Id] = newSubscription;
if(subscriptionTimer == null)
{
subscriptionTimer = new Timer(CheckSubscriptions, null, 5000, 15000);
}
return $"Subscribed. Id: {newSubscription.Id}, Expiration: {newSubscription.ExpirationDateTime}";
}
But I have noticed the timer does not always get triggered(eg: network related issue/ after a fresh deployment of the code).
Is there a better way to replace this timer?
I have heard about webjobs in azure, is it possible to replace this timer with azure webjobs? If so can someone point me to some documentation on how?
I noticed you're using ASPNET, if you're using the ASPNETCORE version, since 2.x, you can have background services that you could run every X hours to resubscribe (update) the subscriptions. That's what we're using (and we're on Azure). I would guess webjobs could do the trick, but I have not used them yet. You could also have an external service that calls one of your endpoints every X hours (like a CRON job) .
hope this helps!
JS
I am using Service Bus Explorer as a quick way of testing a rule that does not work when deployed via ARM.
In JavaScript in the Azure Function I am setting the Topic message to:
context.bindings.outputSbMsg = { Indicator: 'Itinerary'};
In Service Bus Explorer I am setting a Rule on a Subscription with this string:
Indicator = 'Itinerary'
But messages sent to the Topic do not go to this Subscription ( they go to another with the rule 1 = 1)
Question: What am I missing here?
Supplementary info:
I do not seem to have access to the Indicator property. As a test I created an action on the 1=1 rule that appended to the Indicator property and the result was empty.
I am able to access the Indicator property in JavaScript if I have a Function that is triggered by the 1 = 1 rule, so the property is there.
The rule doesn't work because
The rule works against system or user-defined properties rather than message body.
What js function outputs is merely message body, i.e. context.bindings.outputSbMsg = { Indicator: 'Itinerary'}; sends a message { Indicator: 'Itinerary'} and no property is set by us.
And the default rule with 1=1 true filter enables all messages to be selected into the subscription, so you see messages went there all the time. Check doc of topic filters for more details.
For now, it's by design that js function output can't populate message properties. To make the filter work, we have to send messages with property using SDK instead. Install azure-sb package then try sample code below.
const azuresb = require("azure-sb");
const connStr = "ServiceBusConnectionString";
const mytopic = "mytopic";
var serviceBus = azuresb.createServiceBusService(connStr);
const msg =
{
body: "Testing",
customProperties: {
Indicator: 'Itinerary'
}
};
serviceBus.sendTopicMessage(mytopic, msg, function(error) {
if (error) {
context.log(error);
}
else{
context.log("Message Sent");
}
});
We have a job hosted in an azure website, the job reads entries from a topic subscription. Everything works fine when we only have one instance to host the website. Once we scale out to more than one instance we observe the message is processed as many times as instances we have. Each instance points to the same subscription. From what we read, once the item is read, it won't be available for any other process. The duplicated processing is happening inside the same instance, meaning that if we have two instances, the item is processed twice in one of the instances, it is not splitted.
What can be possible be wrong in the way we are doing things?
This is how we proceed to configure the connection to the queue, if the subscription does not exists, it is created:
var serviceBusConfig = new ServiceBusConfiguration
{
ConnectionString = transactionsBusConnectionString
};
config.UseServiceBus(serviceBusConfig);
var allRule1 = new RuleDescription
{
Name = "All",
Filter = new TrueFilter()
};
SetupSubscription(transactionsBusConnectionString,"topic1", "subscription1", allRule1);
private static void SetupSubscription(string busConnectionString, string topicNameKey, string subscriptionNameKey, RuleDescription newRule)
{
var namespaceManager =
NamespaceManager.CreateFromConnectionString(busConnectionString);
var topicName = ConfigurationManager.AppSettings[topicNameKey];
var subscriptionName = ConfigurationManager.AppSettings[subscriptionNameKey];
if (!namespaceManager.SubscriptionExists(topicName, subscriptionName))
{
namespaceManager.CreateSubscription(topicName, subscriptionName);
}
var subscriptionClient = SubscriptionClient.CreateFromConnectionString(busConnectionString, topicName, subscriptionName);
var rules = namespaceManager.GetRules(topicName, subscriptionName);
foreach (var rule in rules)
{
subscriptionClient.RemoveRule(rule.Name);
}
subscriptionClient.AddRule(newRule);
rules = namespaceManager.GetRules(topicName, subscriptionName);
rules.ToString();
}
Example of the code that process the topic item:
public void SendInAppNotification(
[ServiceBusTrigger("%eventsTopicName%", "%SubsInAppNotifications%"), ServiceBusAccount("OutputServiceBus")] Notification message)
{
this.valueCalculator.AddInAppNotification(message);
}
This method is inside a Function static class, I'm using azure web job sdk.
Whenever the azure web site is scaled to more than one instance, all the instances share the same configuration.
It sounds like you're creating a new subscription each time your new instance runs, rather than hooking into an existing one. Topics are designed to allow multiple subscribers to attach in that way as well - usually though each subscriber has a different purpose, so they each see a copy of the message.
I cant verify this from your code snippet but that's my guess - are the config files identical? You should add some trace output to see if your processes are calling CreateSubscription() each time they run.
I think I can access the message id, I'm using azure web job sdk but I think I can find a way to get it. Let me check it and will let you know.