Our application performs several batches of TableBatchOperation. We ensure that each of these table batch operations has
100 or fewer table operations
table operations for one entity partition key only
Along the lines of the following:
foreach (var batch in batches)
{
var operation = new TableBatchOperation();
operation.AddRange(batch.Select(x => TableOperation.InsertOrReplace(x)));
await table.ExecuteBatchAsync(operation);
}
When we use emulated storage we 're hitting a Microsoft.WindowsAzure.Storage.StorageException - "Element 99 in the batch returned an unexpected response code."
When we use production Azure, everything works fine.
Emulated storage is configured as follows:
<add key="StorageConnectionString" value="UseDevelopmentStorage=true;" />
I'm concerned that although everything is working OK in production (where we use real Azure), the fact that it's blowing up with emulated storage may be symptomatic of us doing something we shouldn't be.
I've run it with a debugger (before it blows up) and verified that (as per API):
The entire operation is only only 492093 characters when serialized to JSON (984186 bytes as UTF-16)
There are exactly 100 operations
All entities have the same partition key
See https://learn.microsoft.com/en-us/dotnet/api/microsoft.windowsazure.storage.table.tablebatchoperation?view=azurestorage-8.1.3
EDIT:
It looks like one of the items (#71/100) is causing this to fail. Structurally it is no different to the other items, however it does have some rather long string properties - so perhaps there is an undocumented limitation / bug?
EDIT:
The following sequence of Unicode UTF-16 bytes (on a string property) is sufficent to cause the exception:
r e n U+0019 space
114 0 101 0 110 0 25 0 115 0 32 0
(it's the bytes 25 0 115 0 i.e. unicode end-of-medium U+0019 which is causing the exception).
EDIT:
Complete example of failing entity:
JSON:
{"SomeProperty":"ren\u0019s ","PartitionKey":"SomePartitionKey","RowKey":"SomeRowKey","Timestamp":"0001-01-01T00:00:00+00:00","ETag":null}
Entity class:
public class TestEntity : TableEntity
{
public string SomeProperty { get; set; }
}
Entity object construction:
var entity = new TestEntity
{
SomeProperty = Encoding.Unicode.GetString(new byte[]
{114, 0, 101, 0, 110, 0, 25, 0, 115, 0, 32, 0}),
PartitionKey = "SomePartitionKey",
RowKey = "SomeRowKey"
};
According to your description, I also can reproduce the issue that you mentioned. After I tested I found that the special
Unicode Character 'END OF MEDIUM' (U+0019) seems that not supported by Azure Storage Emulator. If replace to other unicode is possible, please try to use another unicode to instead of it.
we also could give our feedback to Azure storage team.
Related
My SwiftUI segmented control picker uses plain Int ".tag(1)" etc values for its selection.
CoreData only has Int16, Int32 & Int64 options to choose from, and with any of those options it seems my picker selection and CoreData refuse to talk to each other.
How is this (??simple??) task achieved please?
I've tried every numeric based option within CoreData including Int16-64, doubles and floats, all of them break my code or simply just don't work.
Picker(selection: $addDogVM.gender, label: Text("Gender?")) {
Text("Boy ♂").tag(1)
Text("?").tag(2)
Text("Girl ♀").tag(3)
}
I expected any of the 3 CoreData Int options to work out of the box, and to be compatible with the (standard) Int used by the picker.
Each element of a segmented control is represented by an index of type Int, and this index therefore commences at 0.
So using your example of a segmented control with three segments (for example: Boy ♂, ?, Girl ♀), each segment is represented by three indexes 0, 1 & 2.
If the user selects the segmented control that represents Girl ♀, then...
segmentedControl.selectedSegmentIndex = 2
When storing a value using Core Data framework, that is to be represented as a segmented control index in the UI, I therefore always commence with 0.
Everything you read from this point onwards is programmer preference - that is and to be clear - there are a number of ways to achieve the same outcome and you should choose one that best suits you and your coding style. Note also that this can be confusing for a newcomer, so I would encourage patience. My only advice, keep things as simple as possible until you've tested and debugged and tested enough to understand the differences.
So to continue:
The Apple Documentation states that...
...on 64-bit platforms, Int is the same size as Int64.
So in the Core Data model editor (.xcdatamodeld file), I choose to apply an Integer 64 attribute type for any value that will be used as an Int in my code.
Also, somewhere, some time ago, I read that if there is no reason to use Integer 16 or Integer 32, then default to the use of Integer 64 in object model graph. (I assume Integer 16 or Integer 32 are kept for backward compatibility.) If I find that reference I'll link it here.
I could write about the use of scalar attribute types here and manually writing your managed object subclass/es by selecting in the attribute inspector Class Codegen = Manual/None, but honestly I have decided such added detail will only complicate matters.
So your "automatically generated by Core Data" managed object subclass/es (NSManagedObject) will use the optional NSNumber? wrapper...
You will therefore need to convert your persisted/saved data in your code.
I do this in two places... when I access the data and when I persist the data.
(Noting I assume your entity is of type Dog and an instance exists of dog i.e. let dog = Dog())
// access
tempGender = dog.gender as? Int
// save
dog.gender = tempGender as NSNumber?
In between, I use a "temp" var property of type Int to work with the segmented control.
// temporary property to use with segmented control
private var tempGender: Int?
UPDATE
I do the last part a little differently now...
Rather than convert the data in code, I made a simple extension to my managed object subclass to execute the conversion. So rather than accessing the Core Data attribute directly and manipulating the data in code, now I instead use this convenience var.
extension Dog {
var genderAsInt: Int {
get {
guard let gender = self.gender else { return 0 }
return Int(truncating: gender)
}
set {
self.gender = NSNumber(value: newValue)
}
}
}
Your picker code...
Picker(selection: $addDogVM.genderAsInt, label: Text("Gender?")) {
Text("Boy ♂").tag(0)
Text("?").tag(1)
Text("Girl ♀").tag(2)
}
Any questions, ask in the comments.
I'm implementing shouldWaitForLoadingOfRequestedResource handler for AVPlayer/AVURLAsset HLS videos and found a weird behavior in tvOS.
As I see it could request the same resources multiple times, including "root" manifest, second-level manifests and segments (and I'm talking not about multiple quality switches, it's requesting exactly the same resources).
In the same time, each request are served with my code well enough – video is playing good.
Also, exactly the same code is working fine in iOS – no duplicated requests.
In which cases AVURLAsset/AVAssetResourceLoader could request the same resources multiple times in tvOS?
I have the same problem
I can add that I'm using the apple sample app
public func resourceLoader(_ resourceLoader: AVAssetResourceLoader, shouldWaitForLoadingOfRequestedResource loadingRequest: AVAssetResourceLoadingRequest) -> Bool {
print("\(#function) was called in AssetLoaderDelegate with loadingRequest: \(loadingRequest)")
var ret : Bool = true
ret = shouldLoadOrRenewRequestedResource(resourceLoadingRequest: loadingRequest)
return ret
}
This is the debug print i get:
resourceLoader(_:shouldWaitForLoadingOfRequestedResource:) was called
in AssetLoaderDelegate with loadingRequest:
{ URL:
skd://817015000008100f172b492d3b25f5dda31c59d090b21000 }, request
ID = 3, content information request =
AVAssetResourceLoadingContentInformationRequest: 0x14f6dd070, content
type = "(null)", content length = 0, byte range access supported = NO,
disk caching permitted = NO, renewal date = (null), data request =
AVAssetResourceLoadingDataRequest: 0x14f67ae50, requested offset = 0,
requested length = 9223372036854775807, requests all data to end of
resource = YES, current offset = 0
I can see each time the value request ID in the print is different
We want to implement a character counter in our Javascript data entry form, so the user gets immediate keystroke feedback as to how many characters he has typed and how many he has left (something like "25/100", indicating current string length is 25 and 100 is the max allowed).
To do this, I would like to write a service that returns a list of dto property names and their max allowed lengths.
{Name='SmallComment', MaxLength=128}
{Name='BigComment', MaxLength=512}
The best way I can think of to do this would be to create an instance of the validator for that dto and iterate through it to pull out the .Length(min,max) rules. I had other ideas as well, like storing the max lengths in an attribute, but this would require rewriting all the validators to set up the rules based on the attributes.
Whatever solution is best, the goal is to store the max length for each property in a single place, so that changing that length affects the validation rule and the service data passed down to the javascript client.
If you want to maintain a single source of reference for both client/server I would take a metadata approach and provide a Service that returns the max lengths to the client for all types, something like:
public class ValidationMetadataServices : Service
{
public object Any(GetFieldMaxLengths request)
{
return new GetFieldMaxLengthsResponse {
Type1 = GetFieldMaxLengths<Type1>(),
Type2 = GetFieldMaxLengths<Type2>(),
Type3 = GetFieldMaxLengths<Type3>(),
};
}
static Dictionary<string,int> GetFieldMaxLengths<T>()
{
var to = new Dictionary<string,int>();
typeof(T).GetPublicProperties()
.Where(p => p.FirstAttribute<StringLengthAttribute>() != null)
.Each(p => to[p.PropertyName] =
p.FirstAttribute<StringLengthAttribute>().MaximumLength);
return to;
}
}
But FluentValidation uses Static properties so that would require manually specifying a rule for each property that validates against the length from the property metadata attribute.
We have a solution where we parallelize reading and writing to Azure Table Storge.
Because the TableServiceContext does not support reading an entity on one thread and saving it on another thread, we want to keep the Entity using another Context. To do that we need to set:
context.MergeOption = MergeOption.NoTracking;
And when updating (or removing) an entity we call:
context.AttachTo(entitySetName, entity, eTag);
However to do that we need to know the ETag, and I don't know how to get that.
If the entity was tracked, we could use the EntityDesciptor.ETag like this:
private string GetETagFromEntity<T>(T entity) where T : TableServiceEntity
{
return context.Entities.Single(entityDescriptor =>
entityDescriptor.Entity == entity).ETag;
}
... but context.Entities are empty because we don't track entities.
The only solution we found so fare is:
context.AttachTo(entitySetName, entity, "*");
... but that means we have concurrency problems where the last to write always wins.
We also tried to construct the following which works on local Compute Emulator but not in the cloud:
private string GetETagFromEntity<T>(T entity) where T : TableServiceEntity
{
string datePart = entity.Timestamp.ToString("yyyy-MM-dd");
string hourPart = entity.Timestamp.ToString("HH");
string minutePart = entity.Timestamp.ToString("mm");
string secondPart = entity.Timestamp.ToString("ss");
string milisecondPart = entity.Timestamp.ToString("fff").TrimEnd('0');
return string.Format(
"W/\"datetime'{0}T{1}%3A{2}%3A{3}.{4}Z'\"",
datePart,
hourPart,
minutePart,
secondPart,
milisecondPart
).Replace(".Z", "Z");
}
The general problem with this approach even if we could get it to work, is that Microsoft does not make any garanties about how the ETag looks, so this could change over time.
So the question is: How do we get ETag of a Azure Table storage Entity that is not tracked?
I think you'll have to note the etag when you read the entity. (There's probably an event you can hook, maybe ReadingEntity where you can access the etag and store it somewhere.)
I have written an alternate table storage client which is very explicit in exposing the etag and can be used context free and is thread safe. It may work for you. It is available at www.lucifure.com.
I wrote a simpe test to validate my own understanding of the thrift interface for Cassandra. It just inserts a row into the database (using the keyspace and column familty that come preconfigured with the cassandra installation), then reads it from the database and compares the results.
public class CassandraAPITest {
#Test
public void testCassandraAPI() throws Exception {
TTransport tr = new TSocket("localhost", 9160);
tr.open();
Client client = new Cassandra.Client(new TBinaryProtocol(tr));
String key = "123";
byte[] value = { 52, 53, 54 };
ColumnPath columnPath = new ColumnPath("Standard1");
columnPath.setColumn("abc".getBytes("UTF8"));
long timestamp = System.currentTimeMillis();
client.insert("Keyspace1", key, columnPath, value, timestamp, ConsistencyLevel.ONE);
SlicePredicate predicate = new SlicePredicate();
SliceRange sliceRange = new SliceRange();
sliceRange.setStart(new byte[0]);
sliceRange.setFinish(new byte[0]);
predicate.setSlice_range(sliceRange);
List<ColumnOrSuperColumn> result = client.get_slice("Keyspace1", key, new ColumnParent("Standard1"), predicate, ConsistencyLevel.ONE);
assertEquals(1, result.size());
byte[] actual = result.get(0).column.value;
assertArrayEquals(value, actual);
// client.remove("Keyspace1", key, columnPath, System.currentTimeMillis(), ConsistencyLevel.ONE);
tr.close();
}
}
This test runs fine. Of course it leaves a row behind in the database. I could delete the row at the end of the test by uncommenting the client.remove statement above (this also works fine). But what I tried instead was deleting the row via the command-line interface:
cassandra> connect localhost/9160
Connected to: "Test Cluster" on localhost/9160
cassandra> get Keyspace1.Standard1['123']
=> (column=616263, value=456, timestamp=1287909211506)
Returned 1 results.
cassandra> del Keyspace1.Standard1['123']
row removed.
cassandra> get Keyspace1.Standard1['123']
Returned 0 results.
The test fails afterwards. Inserting the row into the database seems to have no effect anymore, so the line assertEquals(1, result.size()) fails:
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:618)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:443)
at org.junit.Assert.assertEquals(Assert.java:427)
at test.package.CassandraAPITest.testCassandraAPI(CassandraAPITest.java:48)
I don't get any error messages (neither on the client nor on the server) and I have no idea what the cause of the problem might be.
You are inserting with millisecond resolution but the CLI (and other high level clients) uses microseconds. So your second insert is in the past, compared to the delete, so Cassandra correctly ignores it.