Azure Table Storage Projection Query and Not Implemented Exception - azure

I have a Person class that I save to a table in Azure Table Storage.
I want to query it with one of the following queries:
var query = from getThis in _serviceContext.CreateQuery<PersonForSearch>(_tableName)
where getThis.Name.Contains(searchTerm)
select new Person
{
PartitionKey = getThis.PartitionKey,
RowKey = getThis.RowKey,
Name = getThis.Name
};
OR
CloudTableQuery<Person> query =
(from getThis in _serviceContext.CreateQuery<Person>(_tableName)
where getThis.Name.Contains(searchTerm)
select getThis).AsTableServiceQuery<Person>();
With either one, I get the following error, thrown on the foreach loop I use to loop through the results of the query:
NotImplemented
The requested operation is not implemented on the specified resource.
I thought that perhaps this resulted from the fact that my Person model does not inherit from TableServiceEntity (I refuse to introduce that coupling - so I decorated it with this attribute instead: [DataServiceKey("PartitionKey", "RowKey")] and manually gave it a PartitionKey and RowKey property.
So I tried to create an entity that DID inherit from TableServiceEntity, which would allow me to query this table (as you can see from the queries, the only property I'm worried about is Name).
This new entity is as follows:
class PersonForSearch : TableServiceEntity
{
public string Name { get; set; }
}
However, this hasn't solved the problem. Is this error talking about some other resource than the class I'm using in my query?

There are a two issues here:
1) Azure Table Storage does not support Contains() method. This is the reason you're getting Not Implemented exception. ATS does support string.Compare() for any range-type operations on strings
2) In order to retrieve the data effectively, you can only search on PartitionKey or on PartitionKey/RowKey combination. Any other query will result in either errors or the download of the full table into client's memory (can't remember which one). If your table is small, download it into memory completely by dropping the 'where' clause and afterwards use Linq for Objects to query however you like. If it is large, find a way to compare against PartitionKey or PartitionKey/RowKey fields
If I'm understanding what you're trying to do correctly, is that you're trying to do a partial string-search through employee table. Overall, ATS is not a very good solution for doing string-based searches (unless these are "starts with" searches on PartitionKey or PartitionKey/RowKey fields). I'd like to highly recommend Lucene.NET for doing text-based searches in the cloud. There is an Azure Directory API for Lucene.NET available as well. Or switch to SQL Azure
HTH

Related

How to query the "Props" table from Flexible Search?

Inside my code I have the primary keys of some page components in Hybris and I have to find all their attributes (properties like title, content etc.)
I can query the "Props" table using an SQL query, but I'm not sure how to find the equivalent in Flexible Search, since querying the database directly is not recommended.
Is there any other simpler way to retrieve all the properties of a component having just the primary key?
What you can do is retrieve the required component using it's primary key. Example:
select {pk} from {SimpleCMSComponent} where {pk} ='8796093056060'
Usually flexible searches are used in DAO classes(Platform example: DefaultProductDao).
When this flexible search is run by the flexible search service you get a ComponentModel. Please see below a groovy script I just created in order to exemplify how to print the ID of a Component retrieved based on its PK(In the same way by using getters you can get the title, content, etc.. ):
import de.hybris.platform.cms2.model.contents.components.SimpleCMSComponentModel;
def flexibleSearchService = spring.getBean("flexibleSearchService");
SimpleCMSComponentModel simpleCmsComponent = flexibleSearchService.search("select {pk} from {SimpleCMSComponent} where {pk} ='8796093056060'").getResult().get(0);
println(simpleCmsComponent.getUid())
I believe though that the best practice in case of CMS Components is to use their IDs instead of PK.

Table storage design for querying with a value within a range

I am looking at using a Table Storage, like Azure or Google or Apache HBase type for storing entities/rows but I could not find any help online for my usage pattern. It goes like this:
Entity has an ID or key, say "UserId"
Date value like "StartedUsingProduct"
Date value like "StoppedUsingProduct"
The queries will mostly be of type "At date T, find all users who were using the product". Note that the same UserId will have many (thousands) of start/stop pairs.
In the Azure case,
- PartitionKey would be UserID
- RowKey "StartedUsingProduct"
but then I can't find a decent way to query without going through a full partition scan.
In the Google case, following their recommendation,
- key would be like "UserID_StartUsingProduct",
and I get the same problem when I have to retrieve a substantial amount of rows and then filter out using the second property.
Has anyone some insights on how to attack this particular usage pattern?
Assuming a very simple entity design considering just these three custom attributes:
UserId
Action - Would indicate either start or stop
DateTime
So when a user starts using the product, you insert an entity for that user with Action = start and DateTime = current date/time. Likewise when a user stops using the product, you insert an entity for that user with Action = stop and DateTime = current date/time.
The pattern that you will have to use is store multiple records for a single activity.
This is needed because you could either be querying on a date (like you mentioned above) or querying on a user e.g. tell me how many times this user has started/stopped using the product.
1st Entity will have PartitionKey = UserId and RowKey = Current Date/Time. You can store the current date/time in form of ticks converted to string using something like DateTimeValue.Ticks.ToString("d20"). This pattern will ensure that you will be able to query activity by the user. You would specify the query as PartitionKey eq UserId and you will get all the records for that user. If you're interested in getting the latest activities first, you could use reverse ticks in RowKey using something like (DateTime.MaxValue.Ticks - DateTimeValue.Ticks).ToString("d20"). This will ensure that the latest records are prepended instead of appended.
2nd Entity will have PartitionKey = DateTimeValue.Date and RowKey = UserId. If you think that a user would start/stop using the software more than one time in a day, you would want to append the date time value in RowKey as well using something like RowKey = UserId|DateTimeValue. This will ensure that multiple start/stop activities for a user in a day can be logged without overwriting the previous activity for that user on that day. Now you can do a query on PartitionKey and that will tell you exactly what all users started/stopped using the product.

Is it possible to create a cross-database query with ServiceStack ORMLite?

Pretty much summed up in the title.
Trying to hit a table in one database and join it to a table in another database on the same server.
I would have assumed an attribute for Database that I could decorate the POCO with, but do not see one that would be appropriate.
Currently using this syntax:
var result = db.Select<Model.Dto>(
db.From<Data.Dto1>()
.Join<Data.Dto2>(d1, d2) => d1.Id == d2.Id));
There's no specific attribute for specifying an external database but in RDBMS's that support cross-database queries you should be able to use the [Schema] attribute, e.g:
[Schema("Server1.Database1.dbo")]
public class Dto { ... }

Linq to Sql using a non-column attribute property in Association attribute

I am trying to create an association between 2 linq to sql entities, say Entity A and Entity B.
A uses a non-column attribute property ( named BaseDocumentType ) and a column attribute in an Association for "ThisKey" and 2 column attributes for "OtherKey". The following is an example of my Association attribute definition...
[System.Data.Linq.Mapping.AssociationAttribute ( ... ThisKey = "BaseDocumentType, Column2" , OtherKey = "Column1,Column2" )]
When I run it I get the following error...
"Data member 'System.String BaseDocumentType' of type 'Library' is not part of the mapping for type 'A'. Is the member above the root of an inheritance hierarchy?"
How can I define the relationship using the non-column attribute property or how do I make this work?
Thanks.
The message is quite clear. LINQ to SQL translates statements to SQL and you tried to use a property that doesn't map to a column, so it can't be translated to SQL.
You'll have to retrieve the entities you want from the database then try to query them using LINQ to Objects, ie LINQ operations on the resulting lists or arrays. A better option is to rethink your design and find a way to retrieve only the data you need from the database and avoid processing the results on the client.
Linq to SQL in this case prevented you from doing something really dangerous. It could have retrieved all the data and process them using your non-column attribute but that would create an enormous performance hit. Some naive LINQ providers actually do just that. Imagine retrieving 1000 objects from the database only to find the two objects that match over this non-column attribute.

How to preform a relative complement query in CRM?

Background (ie what the heck is a relative complement?)
Relative Complement
What I'm trying to do
Let's say I've got a custom Vehicle entity that has a VehicleType option set that is either "Car", or "Truck". There is a 1 to many relationship between Contact and Vehicle (ie. ContactId is on the vehicle entity). How do I write an XRM query (Linq To CRM, QueryExpression, fetch Xml, whatever) that returns the contacts with only cars?
Option 1:
I’d prefer a modification of the proposal that AdamV makes above. I can’t think of a way that you’d get this particular query answered using Linq to CRM, Query Expressions, FetchXML alone. Daryl doesn’t offer what the client is, but I would suppose if Linq and Query Expressions were acceptable offerings, .NET is on the table. Creating aggregate fields containing the count of the related entity on the parent entity (contact in this case) offers more than the Boolean option. If the query requirements ever changed to a threshold (more than X cars, less than Y trucks, between X and Y total vehicles) the Boolean options fails to deliver. The client in this question isn’t known, but I can’t think of many (any?) cases where pulling all the records to the client on a set of 500K+ rows is more efficient than a the SQL query that CRM would make on your behalf against several integer fields with range clauses.
Upside:
Maintains client purity in Query approach
Simple client query
Probably as performant as possible
Downside:
Setups for Aggregate fields
Workflow or plugin to manage the increment and decrement of the aggregate fields
SQL Script for initial load of the aggregates.
Risk that aggregate fields get out of sync (workflow or plugin fails)
Option 2:
If purity within the client isn’t essential, and .NET is on the table – skip the aggregate fields and the setup and just run SQL against the Views. If you don’t want to work with the ADO.NET, a thin ORM like Dapper, Massive, or PetaPOCO can still give you an object model. As Andreas offers in his comment on the OP’s first answer, it seems like something fairly trivial to do in SQL.
Sketching something from top of mind:
SELECT c.*
FROM Contact
WHERE C.Contactid in (
Select contactid
FROM Vehicle v
group by v.contactid , v.type
having v.type = ‘Car’ and count(contactid) > 1
)
AND NOT IN (
Select contactid
FROM Vehicle v
group by v.contactid , v.type
having v.type <> ‘Car’ and count(contactid) > 1
)
Upside:
Much less work
CRM Entities get left alone
Downside:
Depending on the client and/or the application mixing DataAccess methods is a bit kludgy.
Likely less performant than Option 1
Option 3:
Mix and Match: Take the aggregate fields from Option 1. But update them using a scheduled SQL job (or something similar) with a query similar to the initial load job you’d need to write in Option 1
Upside:
Takes most of the work and risk out of Option 1
Keeps all of the performance of Option 1
Downside:
Some will see this as an unsupported feature.
In order to order to perform a true Relative Complement Query you need to be able to perform a subquery.
Your query would basically say give me all the contacts with cars, and then, within those results, remove any contacts that have a vehicle that isn't a car. This is what the SQL in #JasonKoopmans answer does. Unfortunetly, CRM does not support SubQueries.
Therefore, the only way to achieve this is to either perform the sub query on the client side, as I resorted to doing, or storing the results of what would be the subquery in a manner that can be accessed through the main query (ie storing counts on the contact entity).
You could theoretically do this "on the fly" by making a SubQueryResult entity that stores a ContactId, and SubQueryId. You'd first pull back the contacts that have at least 1 car, and create a SubQueryResult record for each record, with it's contactId, and a single SubQueryId that is generated client side to tie them all together.
Then you'd do another query that says give me all the contacts that are in this SubQueryResult with this SubQueryId, that do not have any vehicles that aren't cars.
I could only assume that this wouldn't be any more efficient than performing the two separate queries and performing the filter client side. Although with the new ExecuteMultipleRequests in the new CRM release, it may be close.
I have resorted to pulling back all of my records in CRM, and performing the check on the client side since CRM 2011 doesn't support this via Query Expressions.
You could write two Fetch XML statements, one to return all contacts and the count of their vehicles, and another to return all contacts and the count of their cars, then compare the list on the client side. But once again, you're having to return every contact and filter it client side.
It's not tested but how about this query expression? I'm linking in the Vehicle entity as an inner join, requiring that it's a Car. I'm assuming that the field VehicleType is a String because I'm a bit lazy and don't want to test it (I'm typing this hardcore style, no compilation - pure brain work).
Optionally, you might want to add a Criteria section as well to control which of the Contact instances that actually get retrieved. Do tell how it went!
Sorry for the verbosity. I know you like it short. My brains work better when circumlocutory.
new QueryExpression
{
EntityName = "contact",
ColumnSet = new ColumnSet("fullname"),
LinkEntities =
{
new LinkEntity
{
JoinOperator = JoinOperator.Inner,
LinkFromEntityName = "contact",
LinkFromAttributeName = "contactid",
LinkToEntityName = "vehicle",
LinkToAttributeName = "contactid",
Columns = new ColumnSet("vehicletype"),
EntityAlias = "Vroom",
//LinkCriteria = { Conditions =
//{
// new ConditionExpression(
// "vehicletype", ConditionOperator.Equal, "car")
//} }
LinkCriteria = { Conditions =
{
new ConditionExpression(
"vehicletype", ConditionOperator.NotEqual, "truck")
} }
}
}
};
EDIT:
I've talk to my MVP Gustaf Westerlund and he's suggested the following work-around. Let me stress that it's not an answer to your original question. It's just a way to solve it. And it's cumbersome. :)
So, the hint is to add a flag in the Contact or Person entity. Then, every time you create a new instance of Vehicle, you need to fire a message and using a plugin, update the information on the first about the creation of the latter.
This has several drawbacks.
It requires us to do stuff.
It's not the straight-forward do-this-and-that type of approach.
Maintenance is higher for every new type of Vehicle one adds.
Buggibility is elevated since there are many cases to regard (what happens to the flagification when a Vehicle instance is reasigned, deleted etc.).
So, my answer to your question is changed to: "can't be done". This remains effective until (gladly) proven wrong by presented alternative solution. Duck!
Personally, I'd fetch (almost) everything and unleash the hounds of LINQ onto it. But I'd do that without smiling nor proud. :)

Resources