I want to store data received via topic to Azure tables.
I use Azure functions for that.
public static class Function1
{
[FunctionName("Function1")]
public static async Task Run([ServiceBusTrigger("simple_test_topic", "stt_subscription", AccessRights.Manage, Connection = "ServiceBusConnectionString")]string mySbMsg,
[Table("RoomBasicInformation")] CloudTable outputTable, TraceWriter log)
{
log.Info($"C# ServiceBus topic trigger function processed message: {mySbMsg}");
RoomEntry re = new RoomEntry(){ PartKey = "hpk", RoomId = "13", Temperature = "10"};
var operation = TableOperation.Insert(re);
await outputTable.ExecuteAsync(operation);
}
}
public class RoomEntry : TableEntity
{
public string PartKey { get; set; }
public string RoomId { get; set; }
public string Temperature { get; set; }
}
But after executing it in visual studio I get
mscorlib: Exception while executing function: Function1.
Microsoft.WindowsAzure.Storage: The remote server returned an error:
(400) Bad Request.
You should initialize PartitionKey and RowKey of TableEntity.
Add an entity to a table
Modify RoomEntity to initialize those values:
public class RoomEntry : TableEntity
{
public RoomEntry()
{
}
public RoomEntry(string partitionKey, string roomId)
{
this.PartitionKey = partitionKey;
this.RowKey = roomId;
}
public string RoomId => this.RowKey;
public string Temperature { get; set; }
}
You don't need PartKey property TableEntity already has one for that.
Also modify RommEntity initialization in Azure Function to:
var roomEntry = new RoomEntry("hpk", "13") { Temperature = "10" };
Related
Doing some simple queries using Azure.Data.Table TableClient and the response is not including the ETag.
My entities inherit ITableEntity as required so have the ETag property.
The code is pretty vanilla:
var tableClient = new TableClient(
new Uri(storageUri),
tableName,
new TableSharedKeyCredential(accountName, storageAccountKey));
var response = await tableClient.GetEntityAsync<TEntity>(partitionKey, rowKey);
return response.Value;
The output produced is:
{
"partitionKey": "Partition Key",
"rowKey": "RowKey",
"timestamp": "2023-02-07T20:55:45.6532507+00:00",
"eTag": {},
"forename": "Forename",
"surname": "Surname",
"mobile": "Mobile",
"address": "Address"
}
I'm at a bit of a loss to think what to do as it is all handled within the TableClient so I can't debug all that much.
Any thoughts much appreciated.
UPDATE
I have modified the code to run using the type <TableEntity> and also for the type <User> which is define as:
public class User : ITableEntity, IResult
{
public User() { }
public string PartitionKey { get; set; }
public string RowKey { get; set; }
public DateTimeOffset? Timestamp { get; set; }
public ETag ETag { get; set; }
public string Forename { get; set; }
public string Surname { get; set; }
public string Mobile { get; set; }
public string Address { get; set; }
}
The TableClient GetEntityAsync methods are respectivly:
var response = await tableClient.GetEntityAsync<TableEntity>(partitionKey, rowKey);
var response = await tableClient.GetEntityAsync<User>(partitionKey, rowKey);
The first populates all the fields and the ETag correctly and the latter just populates all of the other fields apart from the ETag.
Dont know if this helps shed some light on what I'm doing wrong?
I tried with some piece of code with Microsoft.azure.cosomos.table to get the Etag.
Code:
using Microsoft.Azure.Cosmos.Table;
namespace azuretablestr
{
class program
{
public static void Main()
{
string connectionString = "your connection string";
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("table1");
TableOperation retrieveOperation = TableOperation.Retrieve<DynamicTableEntity>("partitionkey2", "rowkey2");
TableResult retrievedResult = table.Execute(retrieveOperation);
DynamicTableEntity entity = (DynamicTableEntity)retrievedResult.Result;
string etag = entity.ETag;
Console.WriteLine("Etag :",etag);
}
}
}
Console:
The above code executed successfully and retrieved Etag successfully.
Reference:
Microsoft.Azure.Cosmos.Table Namespace - Azure for .NET Developers | Microsoft Learn
I have a small problem. I am trying to do a migration from Cosmos db to table storage in Azure. The problem is that I haven't dealt with table storage for a long time. It turned out that a large part of the packages that I used before were already out of date. And the offical documentation does not provide the needed information.
Currently I use this method to add item :
public static async Task AddDomain(CloudTable table, DomainEntity domain)
{
TableOperation insertOperation = TableOperation.Insert((Microsoft.WindowsAzure.Storage.Table.ITableEntity)domain);
await table.ExecuteAsync(insertOperation);
}
and this function App to add Domain:
[FunctionName("AddDomain")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function,"post", Route = null)] HttpRequest req,
[Table("Domains")] CloudTable domainsTable,
ILogger log)
{
DomainEntity domain = CreateDomain();
await AddDomain(domainsTable, domain);
return new OkResult();
}
But when i run the function its throw Exception :
Error indexing method 'AddDomain'
and
System.InvalidOperationException at Microsoft.Azure.WebJobs.Extensions.Tables.TableAttributeBindingProvider.TryCreate.
I have same problems with other CRUD operations too. Any idea what is going on ?
Here is how i did it last month
My service class
using Azure;
using Azure.Data.Tables;
using VPMS.FuncApp.EFCore.AuditSetup;
namespace VPMS.Persistence.AuditSetup;
internal class TableStorageService : ITableStorageService
{
private readonly TableServiceClient _tableServiceClient;
public TableStorageService(TableServiceClient tableServiceClient)
{
_tableServiceClient = tableServiceClient;
}
public async Task BulkInsert(string tableName, string partionKey, List<AuditTableStorageTable> audit)
{
var client = _tableServiceClient.GetTableClient(tableName);
List<TableTransactionAction> addEntitiesBatch = new List<TableTransactionAction>();
addEntitiesBatch.AddRange(audit.Select(e => new TableTransactionAction(TableTransactionActionType.Add, e)));
await client.SubmitTransactionAsync(addEntitiesBatch).ConfigureAwait(false);
}
public async Task CreateTableIfNotExistsAsync(string tableName)
{
var client = _tableServiceClient.GetTableClient(tableName);
await client.CreateIfNotExistsAsync();
}
}
My Function App
using Microsoft.Azure.WebJobs;
using Microsoft.EntityFrameworkCore;
using VPMS.FuncApp.EFCore;
using VPMS.FuncApp.EFCore.AuditSetup;
using VPMS.Persistence.AuditSetup;
namespace VPMS.FuncApp.ArchiveAppAuditLogs;
public sealed class ArchiveAppAuditLogTimerTrigger
{
private readonly VPMSDbContext _dbContext;
private readonly ITableStorageService _tableStorageService;
public ArchiveAppAuditLogTimerTrigger(VPMSDbContext dbContext, ITableStorageService tableStorageService)
{
_dbContext = dbContext;
_tableStorageService = tableStorageService;
}
[FunctionName(nameof(ArchiveAppAuditLogTimerTrigger))]
public async Task ArchiveAppAuditLogTrigger([TimerTrigger("%ArchiveAppAuditLogCron%")] TimerInfo myTimer)
{
var currentDate = DateTimeOffset.UtcNow.Date;
var auditTable = _dbContext.Set<Audit>();
await _tableStorageService.CreateTableIfNotExistsAsync(FuncAppConstants.AuditTableName);
var query = auditTable.Where(w => w.CreatedOn < currentDate);
int pageSize = 100;
bool hasMoreRecords = true;
for (int skip = 1; hasMoreRecords is true; skip++)
{
var recordsToSync = await query.Skip(0)
.Take(pageSize)
.ToListAsync();
hasMoreRecords = recordsToSync.Any();
if (hasMoreRecords is false) break;
var groupedAuditData = recordsToSync.GroupBy(b => b.TableName).ToList();
foreach (var groupedAuditDatem in groupedAuditData)
{
List<AuditTableStorageTable> auditStorageTableRecords = new();
foreach (var auditEntry in groupedAuditDatem.ToList()) auditStorageTableRecords.Add(AuditTableStorageTable.BuilData(auditEntry));
await _tableStorageService.BulkInsert(FuncAppConstants.AuditTableName, groupedAuditDatem.Key, auditStorageTableRecords);
}
auditTable.RemoveRange(recordsToSync);
await _dbContext.SaveChangesAsync();
}
}
}
My Azure function app startup class
using Azure.Identity;
using Microsoft.Azure.Functions.Extensions.DependencyInjection;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Azure;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using VPMS.FuncApp.EFCore;
using VPMS.FuncApp.Interfaces;
using VPMS.FuncApp.Services;
using VPMS.Persistence.AuditSetup;
using VPMS.SharedKernel.Interfaces;
[assembly: FunctionsStartup(typeof(VPMS.FuncApp.Startup))]
namespace VPMS.FuncApp;
public sealed class Startup : FunctionsStartup
{
public override void Configure(IFunctionsHostBuilder builder)
{
var configurations = builder.GetContext().Configuration;
builder.Services.AddAzureClients(builder =>
{
builder.AddTableServiceClient(configurations.GetValue<string>("AzureWebJobsStorage"));
builder.UseCredential(new DefaultAzureCredential());
});
}
}
The class the TableStorage inherits from
using Azure;
using Azure.Data.Tables;
using VPMS.Persistence.AuditSetup;
namespace VPMS.FuncApp.EFCore.AuditSetup;
public sealed class AuditTableStorageTable : ITableEntity
{
public string? UserId { get; set; }
public AuditType AuditType { get; set; }
public string TableName { get; set; } = null!;
public DateTimeOffset CreatedOn { get; set; }
public string? OldValues { get; set; }
public string? NewValues { get; set; }
public string? AffectedColumns { get; set; }
public string PrimaryKey { get; set; } = null!;
public Guid BatchId { get; set; }
// these comes from the implemented interface
public string PartitionKey { get; set; }
public string RowKey { get; set; }
public DateTimeOffset? Timestamp { get; set; }
public ETag ETag { get; set; }
public static AuditTableStorageTable BuilData(Audit audit)
{
return new()
{
PartitionKey = audit.TableName,
TableName = audit.TableName,
AffectedColumns = audit.AffectedColumns,
AuditType = audit.AuditType,
BatchId = audit.BatchId,
CreatedOn = audit.CreatedOn,
OldValues = audit.OldValues,
NewValues = audit.NewValues,
PrimaryKey = audit.PrimaryKey,
RowKey = Guid.NewGuid().ToString(),
UserId = audit.UserId,
};
}
}
I am using the new Azure.Data.Tables library from Microsoft to deal with Azure Table Storage. With the old library when you had an entity that implemented ITableEntity and you had a property that you did not want to save to the storage table you would use the [IgnoreProperty] annotation. However, this does not seem to be available on the new library.
What would be the equivalent on the Azure.Data.Tables package or how do you now avoid saving a property to table storage now?
This is the class I want to persist:
public class MySpatialEntity : ITableEntity
{
public int ObjectId { get; set; }
public string Name { get; set; }
public int MonitoringArea { get; set; }
//This is the property I want to ignore because table storage cannot store it
public Point Geometry { get; set; }
//ITableEntity Members
public virtual string PartitionKey { get => MonitoringArea.ToString(); set => MonitoringArea = int.Parse(value); }
public virtual string RowKey { get => ObjectId.ToString(); set => ObjectId = int.Parse(value); }
public DateTimeOffset? Timestamp { get; set; }
public ETag ETag { get; set; }
}
As of version 12.2.0.beta.1, Azure.Data.Tables table entity models now support ignoring properties during serialization via the [IgnoreDataMember] attribute and renaming properties via the [DataMember(Name="<yourNameHere>")] attribute.
See the changelog here.
I don't think there's anything like [IgnoreProperty] available as of now (at least with version 12.1.0).
I found two Github issues which talk about this:
https://github.com/Azure/azure-sdk-for-net/issues/19782
https://github.com/Azure/azure-sdk-for-net/issues/15383
What you can do is create a custom dictionary of the properties you want to persist in the entity and use that dictionary for add/update operations.
Please see sample code below:
using System;
using System.Collections.Generic;
using System.Drawing;
using Azure;
using Azure.Data.Tables;
namespace SO68633776
{
class Program
{
private static string connectionString = "connection-string";
private static string tableName = "table-name";
static void Main(string[] args)
{
MySpatialEntity mySpatialEntity = new MySpatialEntity()
{
ObjectId = 1,
Name = "Some Value",
MonitoringArea = 2
};
TableEntity entity = new TableEntity(mySpatialEntity.ToDictionary());
TableClient tableClient = new TableClient(connectionString, tableName);
var result = tableClient.AddEntity(entity);
}
}
public class MySpatialEntity: ITableEntity
{
public int ObjectId { get; set; }
public string Name { get; set; }
public int MonitoringArea { get; set; }
//This is the property I want to ignore because table storage cannot store it
public Point Geometry { get; set; }
//ITableEntity Members
public virtual string PartitionKey { get => MonitoringArea.ToString(); set => MonitoringArea = int.Parse(value); }
public virtual string RowKey { get => ObjectId.ToString(); set => ObjectId = int.Parse(value); }
public DateTimeOffset? Timestamp { get; set; }
public ETag ETag { get; set; }
public IDictionary<string, object> ToDictionary()
{
return new Dictionary<string, object>()
{
{"PartitionKey", PartitionKey},
{"RowKey", RowKey},
{"ObjectId", ObjectId},
{"Name", Name},
{"MonitoringArea", MonitoringArea}
};
}
}
}
I am trying to resolve list of object using autofac Container, and expecting an empty list in response. However, I am not able to get empty list in return instead getting count as 1.
I also try with without list registration in aotufac conatiner but getting same response.
<pre><code>
class autofacFactory : Autofac.Module
{
protected override void Load(ContainerBuilder builder)
{
builder.RegisterGeneric(typeof(List<>)).As(typeof(IList<>));
builder.RegisterType<Response>().As<IResponse>();
builder.RegisterType<CustomDependencyResolver>().As<ICustomDependencyResolver>();
}
}
public class Response : IResponse
{
public string TransactionNo { get; set; }
public string SchemeCode { get; set; }
}
public interface IResponse
{
string TransactionNo { get; set; }
string SchemeCode { get; set; }
}
public interface ICustomDependencyResolver
{
TResolved Resolve<TResolved>();
}
internal class CustomDependencyResolver : ICustomDependencyResolver
{
private readonly ILifetimeScope _lifetimeScope;
public CustomDependencyResolver(ILifetimeScope lifetimeScope)
{
_lifetimeScope = lifetimeScope;
}
public TResolved Resolve<TResolved>()
=> _lifetimeScope.Resolve<TResolved>();
}
static void Main(string[] args)
{
var builder = new ContainerBuilder();
builder.RegisterModule(new autofacFactory());
using (var container = builder.Build())
{
ICustomDependencyResolver customDependencyResolver = container.Resolve<ICustomDependencyResolver>();
var collection = customDependencyResolver.Resolve<ICollection<IResponse>>();
var list = customDependencyResolver.Resolve<IList<IResponse>>();
}
}
Actual response:
[Image1][1]
[Image2][2]
[Expected Response][3]
[1]: https://i.stack.imgur.com/NVXeW.jpg
[2]: https://i.stack.imgur.com/k58QX.jpg
[3]: https://i.stack.imgur.com/EcFyc.jpg
Try not registering IList<> or List<> - Autofac has built-in support for that.
Can I retrieve the operation DTO from url route inside a service stack service ?
Example :
public class HelloService : IService
{
public object Any(HelloRequest request)
{
//Here I want to retrieve operation Dto.
//In this case if request.AnotherApiRoute is "/another?Age=33"
//then result could be operation AnotherRequest
return new HelloResponse { Result = "Hello, " + name };
}
}
public class AnotherApiService : IService
{
public object Another(AnotherRequest request)
{
return new AnotherResponse { Result = "Your Age : " + Age };
}
}
//OPERATIONS
[Route("/hello/{Name}")]
public class Hello : IReturn<HelloResponse>
{
public string Name { get; set; }
public string AnotherApiRoute {get; set;}
}
public class HelloResponse
{
public string Result { get; set; }
}
[Route("/another/{Age}")]
public class AnotherRequest : IReturn<AnotherResponse>
{
public string Age { get; set; }
}
public class AnotherResponse
{
public string Result { get; set; }
}
Thanks for your suggests
If you want access to the HTTP Request Context the Service was executed in you should inherit from the convenience Service base class (or have your service also implement IRequiresRequestContext so Request is injected), e.g:
public class HelloService : Service
{
public object Any(Hello request)
{
var pathInfo = base.Request.PathInfo;
return new HelloResponse { Result = "Hello, " + name };
}
}
But what you're after is unclear since the Request DTO is the Operation DTO for that request. If instead you wanted to call another Service from within your Service you can do it with Resolving the Service from the IOC (which also injects the current HTTP Request) with:
public class HelloService : Service
{
public object Any(Hello request)
{
using (var service = base.ResolveService<AnotherService>())
{
var anotherDto = request.ConvertTo<Another>();
return service.Any(anotherDto);
}
}
}
Alternatively you can just execute the Service by passing in the Request DTO, and let ServiceStack call the appropriate Service, e.g:
public class HelloService : Service
{
public object Any(Hello request)
{
var anotherDto = request.ConvertTo<Another>();
return base.ExecuteRequest(anotherDto);
}
}