Azure service bus implementation - azure

Basically my ‘Maintopic’ topic receives three types of xml files (‘TEST’,’DEV’,’PROD’).
‘MainSubscription’ subscribes to that topic and now based on the XML file type, I need to route the XML files to:
Respective topics (child topics).
See the below message flow.
Maintopic --à MainSubscription’ (Filter on xml node type)-- > child topic 1 ( xml node type=’TEST’)
child topic 2 ( xml node type=’DEV’)
child topic 3 ( xml node type=’PROD)
I can add a subscription to the ‘Maintopic’, but where can I define all the filter logic for routing the file?
I am new to Azure cloud, how can I do this? I don't even know where to start.

Service Bus supports three filter conditions:
Boolean filters - The TrueFilter and FalseFilter either cause all arriving messages (true) or none of the arriving messages (false) to be selected for the subscription.
SQL Filters - A SqlFilter holds a SQL-like conditional expression that is evaluated in the broker against the arriving messages' user-defined properties and system properties.
Correlation Filters - A CorrelationFilter holds a set of conditions that are matched against one or more of an arriving message's user and system properties.
You must create a filtered subscription which will only receive messages you are interested in.
A filter can be based on any properties of the BrokeredMessage, except the body, since that would require every message to be desterilized in order to be handed on to the correct subscriptions. You can use sql filter.
An example of sql filter is below –
if (!namespaceManager.SubscriptionExists(topicName, filteredSubName1))
{
namespaceManager.CreateSubscription(topicName, filteredSubName1, new SqlFilter("From LIKE '%Smith'"));
}
You don’t send your messages directly to a subscription, you send them to the topic, and that will forward them to all the relevant subscriptions based on their filters. Below is the example -
var message1 = new BrokeredMessage("Second message");
message1.Properties["From"] = "Alan Smith";
var client = TopicClient.CreateFromConnectionString(connectionString, topicName);
client.Send(message1);
Below is how you receive message –
var subClient = SubscriptionClient.CreateFromConnectionString(connectionString, topicName, subscriptionName);
var received = subClient.ReceiveBatch(10, TimeSpan.FromSeconds(5));
foreach (var message in received)
{
Console.WriteLine("{0} '{1}' Label: '{2}' From: '{3}'",
subscriptionName,
message.GetBody<string>(),
message.Label,
message.Properties["From"]);
}

Related

Optimized way of face recognition using Azure Face API

I need to implement face recognition using azure face api . I have developed a programme which is able to find similiar faces using .net SDK . For my use case ,I need to click photo of a person from the webcam and find matching faces from images kept in azure cloud storage . Now, there could be thousand of images in azure cloud storage and in my current implementation of face recognition ,I'm iterating through all the images(kept in azure cloud storage ) and then matching them with the webcam image .
The concern here is :
The face api (provided by azure ) charges 1 dollar per thousand call . Is there a way the search could be optimized such that i don't have to scan the faces which i have already scanned for previous searches
public async Task<List<DetectedFaceAttributes>> FindSimiliarFacesWithAttributesFromContainer(IFaceClient client, string RECOGNITION_MODEL1, string sourceImageFileName)
{
string url = BlobBaseURL;
string sourceurl = sourceContainerURL;
var imagesInNovotraxContainer = await _blobService.GetNames();
IList<Guid?> targetFaceIds = new List<Guid?>();
var faceList = new List<DetectedFaceAttributes>();
// Detect faces from source image url.
IList<DetectedFace> detectedFaces = await DetectFaceRecognize(client, $"{sourceurl}{sourceImageFileName}", RECOGNITION_MODEL1);
if (detectedFaces.Any())
{
foreach (var targetImageFileName in imagesInNovotraxContainer)
{
var faceattribute = new DetectedFaceAttributes();
// Detect faces from target image url.
var faces = await DetectFaceRecognizeWithAttributes(client, $"{url}{targetImageFileName}");
// Add detected faceId to list of GUIDs.
if (faces.Any())
{
targetFaceIds.Add(faces[0].FaceId.Value);
faceattribute.DetectedFace = faces[0];
faceattribute.ImageFileName = targetImageFileName;
faceList.Add(faceattribute);
}
}
// Find a similar face(s) in the list of IDs. Comapring only the first in list for testing purposes.
IList<SimilarFace> similarResults = await client.Face.FindSimilarAsync(detectedFaces[0].FaceId.Value, null, null, targetFaceIds);
var similiarFaceIDs = similarResults.Select(y => y.FaceId).ToList();
var returnDataTypefaceList = faceList.Where(x => similiarFaceIDs.Contains(x.DetectedFace.FaceId.Value)).ToList();
return returnDataTypefaceList;
}
else
{
throw new Exception("no face detected in captured photo ");
}
public async Task<List<DetectedFace>> DetectFaceRecognize(IFaceClient faceClient, string url, string RECOGNITION_MODEL1)
{
// Detect faces from image URL. Since only recognizing, use the recognition model 1.
IList<DetectedFace> detectedFaces = await faceClient.Face.DetectWithUrlAsync(url, recognitionModel: RECOGNITION_MODEL1);
//if (detectedFaces.Any())
//{
// Console.WriteLine($"{detectedFaces.Count} face(s) detected from image `{Path.GetFileName(url)}` with ID : {detectedFaces.First().FaceId}");
//}
return detectedFaces.ToList();
}
Your implementation is not totally clear for me in terms of calls to Face API / your storage (what's behind "DetectFaceRecognizeWithAttributes"). But I think you are right in the fact that you missed something and your global processing is over costly.
What you should do depends on your target:
Is it face "identification"?
Or face "similarity"?
Both have the same logic, but they are using different API operations
Case 1 - Face identification
Process
The global process is the following: you will use a "Person Group" or "Large Person Group" (depending of the number of persons you have) to store data about faces that you already know (the one in you storage), and you will use this group to "identify" a new face. with that, you will do "1-n" search, not "1-1" as you do right now.
Initial setup (group creation):
Choose if you need Person Group or Large Person group, here are the actual limits depending on your pricing:
Person Group:
Free-tier subscription quota: 1,000 person groups. Each holds up to 1,000 persons.
S0-tier subscription quota: 1,000,000 person groups. Each holds up to 10,000 persons.
Large Person Group:
It can hold up to 1,000,000 persons.
Free-tier subscription quota: 1,000 large person groups.
S0-tier subscription quota: 1,000,000 large person groups.
Here I am using Person Group in the explanation, but it's the same methods.
When you know the one you need, create it using "Create" operation.
Then, for each person, you will have to create a "PersonGroup Person" using "PersonGroup Person - Create", and add the corresponding faces to it using "PersonGroup Person - Add Face". Once it is done, you never need to reprocess "detect" operation on those faces.
Then for the "run" part
When you have a new image that you want to compare:
Detect faces in your image with Detect endpoint of Face API
Get the face Ids of your result
Call Identify endpoint of Face API to try to identify those face Ids with your (large) person group
To limit the number of call, you can even do batches of identification calls (up to 10 "input" face Ids in 1 call - see doc).
Case 2 - Face similarity
Here you can use a "Face List" or "Large Face List" to store the faces that you already know, and pass the id of this list when calling "Find Similar" operation. Example with FaceList:
Start with "FaceList - Create" to create your list (doc)
Use "FaceList - Add Face" to add all the faces that you have currently in your blob (doc)
Then for the run, when you call "Find Similar", provide the ID of your FaceList in "faceListId" parameter and the id of the face you want to compare (from Face Detect call)

Logging long JSON gets trimmed in azure application insights

My goal is to log the users requests by using the azure application insights, the requests are being converted into JSON format and then saved.
Sometimes the user request can be very long and it gets trimmed in the azure application insight view which result in not-valid JSON.
Underneath CustomDimensions it looks like:
I'm using the Microsoft.ApplicationInsights.TelemetryClient namespace.
This is my code:
var properties = new Dictionary<string, string>
{
{ "RequestJSON", requestJSON }
};
TelemetryClientInstance.TrackTrace("some description", SeverityLevel.Verbose, properties);
I'm refer this overload:
public void TrackTrace(string message, SeverityLevel severityLevel, IDictionary<string, string> properties);
As per Trace telemetry: Application Insights data model, for Custom Properties, the Max value length is 8192.
In your case, it exceeds the limitation.
I can think of 2 solutions:
1.Write the requestJSON into message field when using TrackTrace method. The trace message Max length is 32768 characters, it may meet your need.
2.Split the requestJSON into more than 1 custom properties, when it's length is larger than 8192. For example, if the length of the requestJSON is 2*8192, then you can add 2 custome properties: property RequestJSON_1 stores the first 8192 data, and property RequestJSON_2 stores the left 8192 data.
When using solution 2, you can easily use Kusto query to join property RequestJSON_1 and property RequestJSON_2 together, so you get the completed / valid json data.

Referencing Message UserProperties from SQLFilter

I'm trying to filter messages for my Subscription Clients in Azure Service Bus. I'd like to check if the property does not exist or the property has a certain value. This is how I add the filter:
var rules = await subscriptionClient_.GetRulesAsync();
if (!rules.Any(r => r.Name == "FilteringRule"))
{
var filter = new SqlFilter($"sys.CustomProperty='{CustomValue}' OR sys.CustomProperty IS NULL");
await subscriptionClient_.AddRuleAsync("FilteringRule", filter);
}
And the same value is passed over to the Microsoft.Azure.ServiceBus.Message object:
Message msg = new Message();
msg.UserProperties.Add("CustomProperty", "CustomValue");
Checking it in Service Bus explorer displays that the message indeed has the property.
I'm trying to figure out how to filter Subscriptions programmatically, not by ARM templates.
The reason the filter is not working because you're trying to treat custom properties as system properties. Please try to change your filter expression to:
var filter = new SqlFilter($"CustomProperty='{CustomValue}' OR CustomProperty IS NULL");
and things should work. Please note that I have removed sys. prefix from your filter expression.

How to discover all Entity Types? One of each?

I need to write a service that connects to CRM, and returns with a list of all of the entity available on the server (custom or otherwise).
How can I do this? To be clear, I am not looking to return all data for all entities. Just a list of every type, regardless of whether any actually exist.
You need to use RetrieveAllEntitiesRequest
RetrieveAllEntitiesRequest request = new RetrieveAllEntitiesRequest()
{
EntityFilters = EntityFilters.Entity,
RetrieveAsIfPublished = true
};
// service is the IOrganizationService
RetrieveAllEntitiesResponse response = (RetrieveAllEntitiesResponse)service.Execute(request);
foreach (EntityMetadata currentEntity in response.EntityMetadata)
{
string logicalName = currentEntity.LogicalName;
// your logic here
}
note that you will get also system or hidden entities, like wizardpage or recordcountsnapshot
You will probably find these sections of the MSDN useful:
Customize Entity Metadata (lookout for the samples linked on that page).
Retrieve and Detect Changes to Metadata.

MQQueue Properties (C# API to Websphere MQ)

I found this code to enumerate a list of queues for a QueueManager.
It works, but I see a lot of System Queues, and even channel names in the list it provides. Is there some property I can test to see if it is a "normal" user-defined queue?
ObjectType, QueueType, Usage seemed to always give same values for every queue-name.
// GET QueueNames - this worked on 07/19/2012 - but returned a lot of system queue, and unclear how to separate user queues from system queues.
PCFMessageAgent agent = new PCFMessageAgent(mqQMgr);
// Build the query request.
PCFMessage requestMessage = new PCFMessage(CMQCFC.MQCMD_INQUIRE_Q_NAMES);
requestMessage.AddParameter(MQC.MQCA_Q_NAME, "*");
// Send the request and retrieve the response.
PCFMessage[] responses = agent.Send(requestMessage);
// Retrieve the values requested from the response.
string[] queueNames = responses[0].GetStringListParameterValue(CMQCFC.MQCACF_Q_NAMES);
//string[] objType = responses[0].GetStringListParameterValue(CMQCFC.MQIACF_OBJECT_TYPE);
int loopCounter = 0;
foreach (string queueName in queueNames)
{
loopCounter++;
Console.WriteLine("QueueName=" + queueName);
try
{
mqQueue = mqQMgr.AccessQueue(
queueName,
MQC.MQOO_OUTPUT // open queue for output
+ MQC.MQOO_INQUIRE // inquire required to get CurrentDepth
+ MQC.MQOO_FAIL_IF_QUIESCING); // but not if MQM stopping
Console.WriteLine("QueueName=" + queueName +
" CurrentDepth=" + mqQueue.CurrentDepth +
" MaxDepth=" + mqQueue.MaximumDepth +
" QueueType=" + mqQueue.QueueType +
" Usage=" + mqQueue.Usage
);
}
catch (MQException mex)
{
Console.WriteLine(mex.Message);
}
}
}
For me your sample code lists only queues, no other objects but yes it lists all queues. You can add another filter requestMessage.AddParameter(MQC.MQIA_Q_TYPE, MQC.MQQT_MODEL); to list only model queues. Other values available for MQC.MQIA_Q_TYPE are MQC.MQQT_LOCAL, MQQT_ALIAS, MQQT_CLUSTER and MQC.MQQT_REMOTE.
All system or predefined queue names begin with SYSTEM. So you could probably use this string filter out predefined queues after listing. Also if you look at a queue definition, there is DEFTYPE attribute, system defined queues have value of PREDEFINED. But I could not add a third parameter to filter queue names by DEFTYPE. I got 3014 reason code.
HTH
As Shashi noted, you will only see queue names from that PCF command.
If you only queue names that begin with PAYROLL then change:
requestMessage.AddParameter(MQC.MQCA_Q_NAME, "*");
to
requestMessage.AddParameter(MQC.MQCA_Q_NAME, "PAYROLL.*");
Or add an if statement to exclude the queue names you do not want to see:
if (!(queueName.startsWith("SYSTEM.")))
{
// do something
}

Resources