I'm trying to get a HashSet<string> into Azure Table Storage. I want to model a project which has members:
public class DbProject : ITableEntity {
public string Name { get; set; } = default!;
public string Owner { get; set; } = default!;
public HashSet<string> Members { get; set; } = default!;
public DateTime CreatedOn { get; set; } = default!;
public string PartitionKey { get; set; } = default!;
public string RowKey { get;set; } = default!;
public DateTimeOffset? Timestamp { get;set; } = default!;
public ETag ETag { get;set; } = default!;
}
The 'insert logic' looks something like:
var dbProject = new DbProject {
Name = projectName,
Owner = owner,
CreatedOn = DateTime.SpecifyKind(DateTime.Now, DateTimeKind.Utc),
Members = new HashSet<string> { owner },
PartitionKey = $"{owner}__{projectName}",
RowKey = $"{owner}__{projectName}"
};
try {
this.tableClient.AddEntity<DbProject>(dbProject);
}
catch (Exception ex) {
Console.WriteLine(ex.Message);
}
The error I'm getting is:
ErrorCode: InvalidInput
Content:
{"odata.error":{"code":"InvalidInput","message":{"lang":"en-US","value":"An error occurred while processing this request.\nRequestId:6019edb7-c002-001c-1be9-bd1379000000\nTime:2021-10-10T15:15:24.5069601Z"}}}
If I remove the HashSet it works like a charm, so I guess there is something wrong with using complex types when creating a record in Azure Table Storage.
I would like to run simple queries like:
public bool IsMember(string owner, string projectName) {
var query = TableClient.CreateQueryFilter<DbProject>(u => u.Name == projectName && u.Members.Contains(owner));
return tableClient.Query<DbProject>(query).Any();
}
Since you are asking for alternative solutions in your comment: The most obvious choice would be to use CosmosDB instead of Table Storage. Table storage is extremely limited in many scenarios. If you don't need to use table storage for a specific reason, using CosmosDB with SQL API is the recommended way for new projects.
As mentioned in Jeremy's comment, you can use HashSet just fine there: How to search on Cosmos DB in a complex JSON Object
If you want to keep Table Storage and don't need to do queries on your HashSet: You can serialize your HashSet to a string like Gaurav mentioned. Maybe something like this:
[IgnoreProperty]
public HashSet<string> Members { get; set; }
public override void ReadEntity(IDictionary<string, EntityProperty> properties, OperationContext operationContext)
{
Members = JsonConvert.DeserializeObject<HashSet<string>>(properties[nameof(Members)].StringValue);
}
public override IDictionary<string, EntityProperty> WriteEntity(OperationContext operationContext)
{
var properties = new Dictionary<string, EntityProperty>();
properties.Add(nameof(Members), new EntityProperty(JsonConvert.SerializeObject(Members)));
return properties;
}
You could also look at this: Adding Complex Properties of a TableEntity to Azure Table Storage
The reason you're running into this issue is because Azure Table Storage does not support complex data types (HashSet is one of them).
For a list of supported data types, please see this link: https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-the-table-service-data-model#property-types.
What you will have to do is somehow serialize the complex data type into one of the supported data types (string data type is the best fit) and save it. When you read the data, you will have to deserialize it back. Please note that in doing so you will lose the capability on querying on this particular attribute.
Related
I am using the new Azure.Data.Tables library from Microsoft to deal with Azure Table Storage. With the old library when you had an entity that implemented ITableEntity and you had a property that you did not want to save to the storage table you would use the [IgnoreProperty] annotation. However, this does not seem to be available on the new library.
What would be the equivalent on the Azure.Data.Tables package or how do you now avoid saving a property to table storage now?
This is the class I want to persist:
public class MySpatialEntity : ITableEntity
{
public int ObjectId { get; set; }
public string Name { get; set; }
public int MonitoringArea { get; set; }
//This is the property I want to ignore because table storage cannot store it
public Point Geometry { get; set; }
//ITableEntity Members
public virtual string PartitionKey { get => MonitoringArea.ToString(); set => MonitoringArea = int.Parse(value); }
public virtual string RowKey { get => ObjectId.ToString(); set => ObjectId = int.Parse(value); }
public DateTimeOffset? Timestamp { get; set; }
public ETag ETag { get; set; }
}
As of version 12.2.0.beta.1, Azure.Data.Tables table entity models now support ignoring properties during serialization via the [IgnoreDataMember] attribute and renaming properties via the [DataMember(Name="<yourNameHere>")] attribute.
See the changelog here.
I don't think there's anything like [IgnoreProperty] available as of now (at least with version 12.1.0).
I found two Github issues which talk about this:
https://github.com/Azure/azure-sdk-for-net/issues/19782
https://github.com/Azure/azure-sdk-for-net/issues/15383
What you can do is create a custom dictionary of the properties you want to persist in the entity and use that dictionary for add/update operations.
Please see sample code below:
using System;
using System.Collections.Generic;
using System.Drawing;
using Azure;
using Azure.Data.Tables;
namespace SO68633776
{
class Program
{
private static string connectionString = "connection-string";
private static string tableName = "table-name";
static void Main(string[] args)
{
MySpatialEntity mySpatialEntity = new MySpatialEntity()
{
ObjectId = 1,
Name = "Some Value",
MonitoringArea = 2
};
TableEntity entity = new TableEntity(mySpatialEntity.ToDictionary());
TableClient tableClient = new TableClient(connectionString, tableName);
var result = tableClient.AddEntity(entity);
}
}
public class MySpatialEntity: ITableEntity
{
public int ObjectId { get; set; }
public string Name { get; set; }
public int MonitoringArea { get; set; }
//This is the property I want to ignore because table storage cannot store it
public Point Geometry { get; set; }
//ITableEntity Members
public virtual string PartitionKey { get => MonitoringArea.ToString(); set => MonitoringArea = int.Parse(value); }
public virtual string RowKey { get => ObjectId.ToString(); set => ObjectId = int.Parse(value); }
public DateTimeOffset? Timestamp { get; set; }
public ETag ETag { get; set; }
public IDictionary<string, object> ToDictionary()
{
return new Dictionary<string, object>()
{
{"PartitionKey", PartitionKey},
{"RowKey", RowKey},
{"ObjectId", ObjectId},
{"Name", Name},
{"MonitoringArea", MonitoringArea}
};
}
}
}
I have a problem in Azure search, I get data from a csv file in blob storage. To simplify let assume my object as:
public class Instrument
{
public Identifier Identifier { get; set; }
[SearchableField(IsSortable = true, IsKey = true)]
public string Id { get; set; }
[SearchableField(IsSortable = true)]
public string RefIs { get; set; }
}
public class Identifier
{
[SearchableField(IsSortable = true)]
public string Code { get; set; }
}
when I receive my data it's flat in the csv such as : _id,_ref,_code
When I create my index through SDK I setup Mapping for simple fields:
indexer.FieldMappings.Add(new FieldMapping("_ref")
{
TargetFieldName = "RefIs"
});
but I cannot figure out the way I declare it to complex type in my indexer ? given code does not work for it :
indexer.FieldMappings.Add(new FieldMapping("_code")
{
TargetFieldName = "Code"
});
indexer.FieldMappings.Add(new FieldMapping("_code")
{
TargetFieldName = "Identifier.Code"
});
error is the same TargetField is not present in index.
Could someone help ?
The best way to overcome is to parse your .csv file your own way and then post documents to ACS using exposed API. This is the best effective way (do not rely on builtin serialiser)
I'm developing an Azure Mobile App service to interface to my Xamarin application.
I've created, connected and successfully populated an SQL Database, but when I try to add some filters to my request, for example an orderby() or where() clauses, it returns me a Bad Request error.
For example, this request: https://myapp.azurewebsites.net/tables/Race?$orderby=iRound%20desc,iYear%20desc&$top=1&ZUMO-API-VERSION=2.0.0 gives me {"message":"The query specified in the URI is not valid. Could not find a property named 'IYear' on type 'MyType'."}.
My configuration method is this:
HttpConfiguration config = new HttpConfiguration();
new MobileAppConfiguration()
.AddTablesWithEntityFramework()
.ApplyTo(config);
config.MapHttpAttributeRoutes();
Database.SetInitializer(new CreateDatabaseIfNotExists<MainDataContext>());
app.UseWebApi(config);
and my DbContext is this:
public class MainDataContext : DbContext
{
private const string connectionStringName = "Name=MS_TableConnectionString";
public MainDataContext() : base(connectionStringName)
{
Database.Log = s => WriteLog(s);
}
public void WriteLog(string msg)
{
System.Diagnostics.Debug.WriteLine(msg);
}
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Conventions.Add(
new AttributeToColumnAnnotationConvention<TableColumnAttribute, string>(
"ServiceTableColumn", (property, attributes) => attributes.Single().ColumnType.ToString()));
}
public DbSet<Race> Race { get; set; }
public DbSet ...ecc...
}
Following this guide, I added a migration after creating my TableControllers. So the TableController for the example type shown above is pretty standard:
[EnableQuery(AllowedQueryOptions = AllowedQueryOptions.All)]
public class RaceController : TableController<Race>
{
protected override void Initialize(HttpControllerContext controllerContext)
{
base.Initialize(controllerContext);
MainDataContext context = new MainDataContext();
DomainManager = new EntityDomainManager<Race>(context, Request);
}
// GET tables/Race
[EnableQuery(AllowedQueryOptions = AllowedQueryOptions.All)]
public IQueryable<Race> GetAllRace()
{
return Query();
}
// GET tables/Race/48D68C86-6EA6-4C25-AA33-223FC9A27959
public SingleResult<Race> GetRace(string id)
{
return Lookup(id);
}
// PATCH tables/Race/48D68C86-6EA6-4C25-AA33-223FC9A27959
public Task<Race> PatchRace(string id, Delta<Race> patch)
{
return UpdateAsync(id, patch);
}
// POST tables/Race
public async Task<IHttpActionResult> PostRace(Race item)
{
Race current = await InsertAsync(item);
return CreatedAtRoute("Tables", new { id = current.Id }, current);
}
// DELETE tables/Race/48D68C86-6EA6-4C25-AA33-223FC9A27959
public Task DeleteRace(string id)
{
return DeleteAsync(id);
}
}
As you can see, I already tried to add the EnableQuery attribute to my TableController, as seen on Google. I also tried to add these filters to the HttpConfiguration object, without any success:
config.Filters.Add(new EnableQueryAttribute
{
PageSize = 10,
AllowedArithmeticOperators = AllowedArithmeticOperators.All,
AllowedFunctions = AllowedFunctions.All,
AllowedLogicalOperators = AllowedLogicalOperators.All,
AllowedQueryOptions = AllowedQueryOptions.All
});
config.AddODataQueryFilter(new EnableQueryAttribute
{
PageSize = 10,
AllowedArithmeticOperators = AllowedArithmeticOperators.All,
AllowedFunctions = AllowedFunctions.All,
AllowedLogicalOperators = AllowedLogicalOperators.All,
AllowedQueryOptions = AllowedQueryOptions.All
});
I don't know what to investigate more, as things seems to be changing too fast for a newbie like me who's first got into Azure.
EDIT
I forgot to say that asking for the complete table, so for example https://myapp.azurewebsites.net/tables/Race?ZUMO-API-VERSION=2.0.0, returns correctly the entire dataset. The problem occurs only when adding some clauses to the request.
EDIT 2
My model is like this:
public class Race : EntityData
{
public int iRaceId { get; set; }
public int iYear { get; set; }
public int iRound { get; set; }
ecc..
}
and the database table that was automatically created is this, including all the properties inherited from EntityData:
Database table schema
Digging into the source code, Azure Mobile Apps sets up camelCase encoding of all requests and responses. It then puts them back after transmission accordign to rules - so iRaceId becomes IRaceId on the server.
The easiest solution to this is to bypass the auto-naming and use a JsonProperty attribute on each property within your server-side DTO and client-side DTO so that they match and will get encoding/decoded according to your rules.
So:
public class Race : EntityData
{
[JsonProperty("id")]
public string Id { get; set; }
[JsonProperty("raceId")]
public int iRaceId { get; set; }
[JsonProperty("year")]
public int iYear { get; set; }
[JsonProperty("round")]
public int iRound { get; set; }
etc..
}
I want my repsository to be independent of the data access technology. Currently I am working on a Xamrin.Forms App that uses Azure Mobile App Services for data access. For performance and flexibility reasons I want my repository to look simmilar like the following:
Task<IEnumerable<IDomainObject>> GetDomainObjectAsync(Func<IQueryable<IDomainObject>, IQueryable<IDomainObject>> query)
Suppose my IDomainObject looks like the following:
public interface IDomainObject
{
string Name { get; }
}
and my DataAccess Object:
internal class AzureDomainObject : IDomainObject
{
public string Name { get; set; }
public string Id { get; set; }
}
As far as I found out and tested I can do the following to query the database within my repository implementation:
public async Task<IEnumerable<IDomainObject>> GetDomainObjectAsync(Func<IQueryable<IDomainObject>, IQueryable<IDomainObject>> query)
{
// _table of type IMobileServiceTable<AzureDomainObject> gotten by MobileServiceClient
var tableQuery = _table.GetQuery();
tableQuery.Query = tableQuery.Query.Take(4); // 1) this was for testing and it works (ordering etc also works)
// tableQuery.Query = query(tableQuery.Query); // 2) this was my initial idea how to use the input param
await _table.ReadAsync(tableQuery);
}
My poblem now is how to use the input param query to replace 1) with 2).
tableQuery.Query expects an IQueryable<AzureDomainObject> but query is of type IQueryable<IDomainObject>.
Neither .Cast<AzureDomainObject>() nor .OfType<AzureDomainObject>() work to convert. Nor does (IQueryable<IAzureDomainObject>)query; work.
Cast and OfType throw NotSupportedException and the hard cast throws an InvalidCastException.
I also tried to extract the Expression from the query input param and assign it to the tableQuery.Query. But then a runtime exception occurs that they are not compatible.
Another idea I had was to use the ReadAsync(string) overload and pass the string representation of the passed query param. But this way I don't know how to generate the string.
So the final question is: Does anyone knows how to hide both AzureDomainObject and IMobileServiceTable from the domain model but keep the flexibility and performance benefits of IQueryable in the repository interface?
According to your description, I checked this issue and here is my implementation for this scenario, you could refer to them.
Model:
public class TodoItem : IDomainObject
{
public string Id { get; set; }
[JsonProperty(PropertyName = "text")]
public string Text { get; set; }
[JsonProperty(PropertyName = "complete")]
public bool Complete { get; set; }
}
public interface IDomainObject
{
string Id { get; set; }
}
Repository:
public interface IAzureCloudTableRepository<T> where T : IDomainObject
{
Task<IEnumerable<T>> GetDomainObjectAsync(Func<IQueryable<T>, IQueryable<T>> query);
}
public class AzureCloudTableRepository<T> : IAzureCloudTableRepository<T> where T : IDomainObject
{
IMobileServiceTable<T> table;
public AzureCloudTableRepository(MobileServiceClient client)
{
this.table = client.GetTable<T>();
}
public async Task<T> CreateItemAsync(T item)
{
await table.InsertAsync(item);
return item;
}
public async Task<IEnumerable<T>> GetDomainObjectAsync(Func<IQueryable<T>, IQueryable<T>> query)
{
var tableQuery = this.table.CreateQuery();
tableQuery.Query = tableQuery.Query.Take(4); //the internal fixed query
tableQuery.Query = query(tableQuery.Query); //the external query
return await tableQuery.ToEnumerableAsync();
}
}
TEST:
var mobileService = new MobileServiceClient("https://{your-app-name}.azurewebsites.net");
var todoitem = new AzureCloudTableRepository<TodoItem>(mobileService);
var items = await todoitem.GetDomainObjectAsync((query) =>
{
return query.Where(q => q.Text!=null);
});
We're switching over from Lokad to CloudFx to handle putting things in/out of table storage.
With Lokad, we had 3 entities:
1) Object 1, which added partition key and row key to
2) Object 2 which had a bunch of properties as well as an instance of
3) Object 3, which had its own set of properties
As far as I can tell, the only way for CloudFx to input that info into table storage is to flatten that whole thing out with one massive object that has all the properties of the previous three objects. With Lokad, we could just useSkinny=true.
Thoughts?
This is the method we ended up implementing. Short version: serialize object 2 and stuff it into the Value property of Object 1, then put in table storage.
Object 1:
public class CloudEntity<T>
{
public CloudEntity()
{
Timestamp = DateTime.UtcNow;
}
public string RowKey { get; set; }
public string PartitionKey { get; set; }
public DateTime Timestamp { get; set; }
public T Value { get; set; }
}
Object 2:
public class Store
{
public string StoreId { get; set; }
public string StoreName { get; set; }
public string StoreType { get; set; }
public Address MailingAddress { get; set; }
public PhoneNumber TelephoneNumber { get; set; }
public StoreHours StoreHours { get; set; }
}
Object 3 can be whatever...the address in this case perhaps...it all gets serialized.
So in code you can get the table as follows (more than 1 way to do this):
var tableStorage = new ReliableCloudTableStorage(connection string you're using);
Then let's say you have an instance of store() you want to put in table storage:
var myStore = new Store(
{
storeId = "9832",
storeName = "Headquarters"
...
});
You can do so in the following way:
var cloudEntity = new CloudEntity<string>
{
PartitionKey = whatever you want your partition key to be,
RowKey = whatever you want your row key to be,
Value = JsonConvert.SerializeObject(myStore) // THIS IS THE MAGIC
};
tableStorage.Add<CloudEntity<string>>(name of table in table storage, cloudEntity);
The entity put in table storage will have all the properties of the CloudEntity class (row key, partition key, etc) and in the "Value" column there will be the json of the object you wanted to store. It's easily readable through Azure Storage Explorer which is nice too.
To get the object back out, use something like this:
var cloudEntity = tableStorage.Get<CloudEntity<string>>(name of your table, partitionKey: whatever the partition key is);
Then you can deserialize the "Value" field of those into the object you're expecting:
var myStore = JsonConvert.DeserializeObject<Store>(cloudEntity.Value);
If you're getting a bunch back, just make myStore a list and loop through the cloudEntities, deserializing and adding each to your list.
Hope this helps!