I have a Cosmos Document DB with many items in it and I'd like to update a single item.
I've tried doing it like this:
const { resources: updatedItem } = await container.item(existingType).replace(newItem)
But get this error:
Illegal characters ['/', '\', '?', '#'] cannot be used in resourceId
Upon further research it's because the _rid is being read from the existingType and contains patterns like AAAAA==/
What is the best practice for dealing with this?
According to the API document
Read the item's definition. Any provided type, T, is not necessarily
enforced by the SDK. You may get more or less properties and it's up
to your logic to enforce it. If the type, T, is a class, it won't pass
typeof comparisons, because it won't have a match prototype. It's
recommended to only use interfaces.
There is no set schema for JSON items. They may contain any number of
custom properties.
You can create entity class so that you can remove system properties.
Related
I am following the below blog which explains how to create operator and import another CR into existing one.
http://heidloff.net/article/accessing-third-party-custom-resources-go-operators/
here https://github.com/nheidloff/operator-sample-go/blob/aa9fd15605a54f712e1233423236bd152940f238/operator-application/controllers/application_controller.go#L276 , spec is created with hardcoded properties.
I want to import the spark operator types in my operator.
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/pkg/apis/sparkoperator.k8s.io/v1beta2/types.go
This spark operator is having say - 100+ types/properties. By following the above blog , i could create the Go object but it would be hardcoded. I want to create the dynamic object based on user provided values in CR YAML. e.g. - customer can provided 25 attributes , sometimes 50 for spark app. I need to have dynamic object created based on user YAML. Can anybody please help me out ?
If you set the spec type to be a json object, you can have the Spec contain arbitrary json/yaml. You don't have to have a strongly typed Spec object, your operator can then decode it and do whatever you want with it during your reconcile operation as long as its you as you can serialize and deserialize it from json. Should be able to set it to json.RawMesage I think?
What do you mean by hardcoded properties?
If I understood it correctly, you want to define an API for a resource which uses both types from an external operator and your customs. You can extend your API using the types from specific properties such as ScheduledSparkApplicationSpec from this. Here is an example API definition in Go:
type MyKindSpec struct {
// using external third party api (you need to import it)
SparkAppTemplate v1beta2.ScheduledSparkApplicationSpec `json:"sparkAppTemplate,omitempty"`
// using kubernetes core api (you need to import it)
Container v1.Container `json:"container,omitempty"`
// using custom types
MyCustomType MyCustomType `json:"myCustomType,omitempty"`
}
type MyCustomType struct {
FirstField string `json:"firstField,omitempty"`
SecondField []int `json:"secondField,omitempty"`
}
I have a Cosmos DB with a container that contains document with varying structure.
I am using the Java SQL API for reading the documents from this container.
The issue I am having is that the API methods for querying/reading the container expects a model class as input param and will return instances of the model class. Because my container contains documents that have varying fields and depth, it is not possible for me to create a model class to represent this.
I need to be able to read/query the documents and then parse it myself and extract the values that I am looking for.
Any ideas? I have used "Object" in the API methods for e.g. queryItem and then it returns a LinkedHashMap that I can parse myself. Is this the way to do it? It looks a bit "raw" but I have not found a better way.
Below is a typical example from the SDK doc. I cannot create a "Family" model class in my code, because the structure can vary from document to document - both which fields are stored and the depth.
private void queryItems() {
CosmosQueryRequestOptions queryOptions = new CosmosQueryRequestOptions();
queryOptions.setQueryMetricsEnabled(true);
CosmosPagedIterable<Family> familiesPagedIterable = container.queryItems(
"SELECT * FROM Family WHERE Family.lastName IN ('Andersen', 'Wakefield', 'Johnson')", queryOptions, Family.class);
familiesPagedIterable.iterableByPage(10).forEach(cosmosItemPropertiesFeedResponse -> {
logger.info("Got a page of query result with {} items(s) and request charge of {}",
cosmosItemPropertiesFeedResponse.getResults().size(), cosmosItemPropertiesFeedResponse.getRequestCharge());
logger.info("Item Ids {}", cosmosItemPropertiesFeedResponse
.getResults()
.stream()
.map(Family::getId)
.collect(Collectors.toList()));
});
}
Per my understanding, it's determined by the sdk funtion's input parameters and output data type. And exactly, we can find that both sample code for java or spring are depends on the data model. So it's really good for you to use Object in your code because of the various documents.
And it's true that we can't design a data model to contain all the properties in the documents but I think it's also a good idea to set a model which contains all the properties required. I mean that maybe you have a useless property in a query, so the query model should exclude it.
I think I found the proper solution:
Create model class. Define the members with unknown depth and structure as JsonNode.
Then the model class could be used and the values of the JsonNode accessed using nice methods.
So I have a Nested Many Schema (eg Users) inside another Schema (eg Computer). My input object to be deserialised by the Schema is complex and does not allow for assignment, and to modify it to allow for assignment is impractical.
The input object (eg ComputerObject) itself does not contain an a value called "Users", but nested in a few other objects is a function that can get the users (eg ComputerObject.OS.Accounts.getUsers()), and I want the output of that function to be used as the value for that field in the schema.
Two possible solutions exist that I know of, I could either define the field as field.Method(#call the function here) or I could do a #post_dump function to call the function and add it to the final output JSON as it can provide both the initial object and the output JSON.
The issue with both of these is that it then doesn't serialise it through the nested Schema Users, which contains more nested Schemas and so on, it would just set that field to be equal to the return value of getUsers, which I don't want.
I have tried to define it in a pre-dump so that it can then be serialised in the dump (note: this schema is used only for dumping and not for loading), but as that takes in the initial object I cannot assign to it.
Basically, I have a thing I am trying to do, and a bunch of hacky workarounds that could make it work but not without breaking other things or missing out on the validation altogether, but no actual solution it seems, anybody know how to do this properly?
For further info, the object that is being input is a complex Django Model, which might give me some avenues Im not aware of, my Django experience is somewhat lacking.
So figured this out myself eventually:
Instead of managing the data-getting in the main schema, you can define the method used in the sub-schema using post_dump with many=True, thus the following code would work correctly:
class User(Schema):
id = fields.UUID
#pre_dump(pass_many=True)
def get_data(self, data, **kwargs):
data = data.Accounts.getUsers()
return data
class Computer(Schema):
#The field will need to be called "OS" in order to correctly look in the "OS" attribute for further data
OS = fields.Nested(User, many=True, data_key="users")
After spending a year working with the Microsoft.Xrm.Sdk namespace, I just discovered yesterday the Entity.FormattedValues property contains the text value for Entity specific (ie Local) Option Set texts.
The reason I didn't discover it before, is there is no early bound method of getting the value. i.e. entity.new_myOptionSet is of type OptionSetValue which only contains the int value. You have to call entity.FormattedValues["new_myoptionset"] to get the string text value of the OptionSetValue.
Therefore, I'd like to get the crmsrvcutil to auto-generate a text property for local option sets. i.e. Along with Entity.new_myOptionSet being generated as it currently does, Entity.new_myOptionSetText would be generated as well.
I've looked into the Microsoft.Crm.Services.Utility.ICodeGenerationService, but that looks like it is mostly for specifying what CodeGenerationType something should be...
Is there a way supported way using CrmServiceUtil to add these properties, or am I better off writing a custom app that I can run that can generate these properties as a partial class to the auto-generated ones?
Edit - Example of the code that I would like to be generated
Currently, whenever I need to access the text value of a OptionSetValue, I use this code:
var textValue = OptionSetCache.GetText(service, entity, e => e.New_MyOptionSet);
The option set cache will use the entity.LogicalName, and the property expression to determine the name of the option set that I'm asking for. It will then query the SDK using the RetrieveAttriubteRequest, to get a list of the option set int and text values, which it then caches so it doesn't have to hit CRM again. It then looks up the int value of the New_MyOptionSet of the entity and cross references it with the cached list, to get the text value of the OptionSet.
Instead of doing all of that, I can just do this (assuming that the entity has been retrieved from the server, and not just populated client side):
var textValue = entity.FormattedValues["new_myoptionset"];
but the "new_myoptionset" is no longer early bound. I would like the early bound entity classes that gets generated to also generate an extra "Text" property for OptionSetValue properties that calls the above line, so my entity would have this added to it:
public string New_MyOptionSetText {
return this.GetFormattedAttributeValue("new_myoptionset"); // this is a protected method on the Entity class itself...
}
Could you utilize the CrmServiceUtil extension that will generate enums for your OptionSets and then add your new_myOptionSetText property to a partial class that compares the int value to the enums and returns the enum string
Again, I think specifically for this case, getting CrmSvcUtil.exe to generate the code you want is a great idea, but more generally, you can access the property name via reflection using an approach similar to the accepted answer # workarounds for nameof() operator in C#: typesafe databinding.
var textValue = entity.FormattedValues["new_myoptionset"];
// becomes
var textValue = entity.FormattedValues
[
// renamed the class from Nameof to NameOf
NameOf(Xrm.MyEntity).Property(x => x.new_MyOptionSet).ToLower()
];
The latest version of the CRM Early Bound Generator includes a Fields struct that that contains the field names. This allows accessing the FormattedValues to be as simple as this:
var textValue = entity.FormattedValues[MyEntity.Fields.new_MyOptionSet];
You could create a new property via an interface for the CrmSvcUtil, but that's a lot of work for a fairly simple call, and I don't think it justifies creating additional properties.
I have an entity that, in addition to a few common properties, contains a list of extended properties stored as (Name, Value) pairs of strings within a collection. I should probably mention that these extended properties widely vary from instance to instance, and that they only need to be listed for each instance (there won't be any queries over the extended properties, for example finding all instances with a particular (Name, Value) pair). I'm exploring how I might persist this entity using Windows Azure Table Services. With the particular approach I'm testing now, I'm concerned that there may be a degradation of performance over time as more distinct extended property names are encountered by the application.
If I were storing this entity in a typical relational database, I'd probably have two tables to support this schema: the first would contain the entity identifier and its common properties, and the second would reference the entity identifier and use EAV style row-modeling to store the extended (Name, Value) pairs, one to each row.
Since tables in Windows Azure already use an EAV model, I'm considering custom serialization of my entity so that the extended properties are stored as though they were declared at compile time for the entity. I can use the Reading- and Writing-Entity events provided by DataServiceContext to accomplish this.
private void OnReadingEntity(object sender, ReadingWritingEntityEventArgs e)
{
MyEntity Entry = e.Entity as MyEntity;
if (Entry != null)
{
XElement Properties = e.Data
.Element(Atom + "content")
.Element(Meta + "properties");
//select metadata from the extended properties
Entry.ExtendedProperties = (from p in Properties.Elements()
where p.Name.Namespace == Data && !IsReservedPropertyName(p.Name.LocalName) && !string.IsNullOrEmpty(p.Value)
select new Property(p.Name.LocalName, p.Value)).ToArray();
}
}
private void OnWritingEntity(object sender, ReadingWritingEntityEventArgs e)
{
MyEntity Entry = e.Entity as MyEntity;
if (Entry != null)
{
XElement Properties = e.Data
.Element(Atom + "content")
.Element(Meta + "properties");
//add extended properties from the metadata
foreach (Property p in (from p in Entry.ExtendedProperties
where !IsReservedPropertyName(p.Name) && !string.IsNullOrEmpty(p.Value)
select p))
{
Properties.Add(new XElement(Data + p.Name, p.Value));
}
}
}
This works, and since I can define requirements for extended property names and values, I can ensure that they conform to all the standard requirements for entity properties within a Windows Azure Table.
So what happens over time as the application encounters thousands of different extended property names?
Here's what I've observed within the development storage environment:
The table container schema grows with each new name. I'm not sure exactly how this schema is used (probably for the next point), but obviously this xml document could grow quite large over time.
Whenever an instance is read, the xml passed to OnReadingEntity contains elements for every property name ever stored for any other instance (not just the ones stored for the particular instance being read). This means that retrieval of an entity will become slower over time.
Should I expect these behaviors in the production storage environment? I can see how these behaviors would be acceptable for most tables, as the schema would be mostly static over time. Perhaps Windows Azure Tables were not designed to be used like this? If so, I will certainly need to change my approach. I'm also open to suggestions on alternate approaches.
Development storage uses SQL Express to simulate cloud table storage. Ignore what you see there... the production storage system doesn't store any schema, so there's no overhead to having lots of unique properties in a table.