I am getting below error while trying to insert multiple entities in Azure Table storage:
com.microsoft.azure.storage.table.TableServiceException: Bad Request
at com.microsoft.azure.storage.table.TableBatchOperation$1.postProcessResponse(TableBatchOperation.java:525)
at com.microsoft.azure.storage.table.TableBatchOperation$1.postProcessResponse(TableBatchOperation.java:433)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:146)
Below is the Java code for batch insert:
public BatchInsertResponse batchInsert(BatchInsertRequest request){
BatchInsertResponse response = new BatchInsertResponse();
String erpName = request.getErpName();
HashMap<String,List<TableEntity>> tableNameToEntityMap = request.getTableNameToEntityMap();
HashMap<String,List<TableEntity>> errorMap = new HashMap<String,List<TableEntity>>();
HashMap<String,List<TableEntity>> successMap = new HashMap<String,List<TableEntity>>();;
CloudTable cloudTable=null;
for (Map.Entry<String, List<TableEntity>> entry : tableNameToEntityMap.entrySet()){
try {
cloudTable = azureStorage.getTable(entry.getKey());
} catch (Exception e) {
e.printStackTrace();
}
// Define a batch operation.
TableBatchOperation batchOperation = new TableBatchOperation();
List<TableEntity> value = entry.getValue();
for (int i = 0; i < value.size(); i++) {
TableEntity entity = value.get(i) ;
batchOperation.insertOrReplace(entity);
if (i!=0 && i % batchSize == 0) {
try {
cloudTable.execute(batchOperation);
batchOperation.clear();
} catch (Exception e) {
e.printStackTrace();
}
}
}
try {
cloudTable.execute(batchOperation);
} catch (Exception e) {
e.printStackTrace();
}
}
}
Above code is working fine if I will assign batchSize value to 10 but if I will assign to 1000 or 100 it will throw Bad request error.
Please help me to resolve this error. I am using Spring boot and Azure-storage Java SDK version 4.3.0.
As Aravind mentioned, 400 error usually means there's something wrong with your data. From this link, an entity batch transaction will fail if one or more of the following conditions are not met:
All entities subject to operations as part of the transaction must have the same PartitionKey value.
An entity can appear only once in the transaction, and only one operation may be performed against it.
The transaction can include at most 100 entities, and its total payload may be no more than 4 MB in size.
All entities are subject to the limitations described in Understanding the Table Service Data Model.
Please check your entities against these four rules and ensure that you're not violating one of the rules.
Related
I am trying to update an item in was dynamoDB using nodes, db.updateItem(query).
I am getting the following error :
Invalid UpdateExpression: Expression size has exceeded the maximum allowed size dynamodb
On reading few posts, I realised that dynamoDB allows itemSize to be 400KB and that might be a problem here. But if that is the problem, why did it allow to insert the item in the first place.
I am not sure what exactly the issue. Any help would be appreciated.
Please let me know if I missed any required information
You are probably hitting Expression Parameters limits. Please refer to:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-expression-parameters
If you are getting this exception
software.amazon.awssdk.services.dynamodb.model.DynamoDbException: Item size has exceeded the maximum allowed size
This exception is due to AWS Dynamodb limits mentioned here
in my case, I compressed the record using gzip and stored binary zipped data, and uncompressed it back after reading that record.
please see below sample code to compress and decompress (I am using enhanced dynamodb client library)
public CompletableFuture<Boolean> storeItem(MyBeanClass object) {
CompletableFuture<Boolean> completableFuture = CompletableFuture.supplyAsync(() -> false);
try {
byte[] serialized = objectMapper.writeValueAsString(object.getLargeData()).getBytes(UTF_8);
if (serialized.length >= 10000) { //large record, gzip it
try (ByteArrayOutputStream bos = new ByteArrayOutputStream(serialized.length);
GZIPOutputStream gzip = new GZIPOutputStream(bos)) {
gzip.write(serialized);
gzip.close();
MyBeanClass newObject = new MyBeanClass();
newObject.setPrimaryId(object.getPrimaryId());
newObject.setGzData(SdkBytes.fromByteArray(bos.toByteArray()));
completableFuture = enhancedDynamoDbTable.putItem(newObject)
.thenApply(res -> true)
.exceptionally(th -> {
th.printStackTrace();
return false;
});
}
} else { //no compression required
completableFuture = enhancedDynamoDbTable.putItem(object).thenApply(res -> true)
.exceptionally(th -> {
th.printStackTrace();
return false;
});
}
} catch (IOException e) {
e.printStackTrace();
}
To fetch record and unzip
public CompletableFuture<MyBeanClass> getItem(String id) {
return enhancedDynamoDbTable
.getItem(Key.builder().partitionValue(id).build())
.thenApply(record -> {
if (record.getGzData() != null) {
try (ByteArrayInputStream arrayInputStream = new ByteArrayInputStream(record.getGzData().asByteArray());
GZIPInputStream inputStream = new GZIPInputStream(arrayInputStream);
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream()) {
byte[] buffer = new byte[1024];
int length;
while ((length = inputStream.read(buffer)) != -1) {
byteArrayOutputStream.write(buffer, 0, length);
}
record = objectMapper.readValue(byteArrayOutputStream.toString(UTF_8), MyBeanClass.class);
} catch (IOException e) {
e.printStackTrace();
}
}
return record;
});
}
Hope that helps.
I have implemented the Azure - Offline Sync based on the documentation / Sample provided by Microsoft Sample in my Xamarin Forms Application.
In the sample / documentation provided, they are using the default Service Handler.
// Simple error/conflict handling. A real application would handle the various errors like network conditions,server conflicts and others via the IMobileServiceSyncHandler.
Since I need to implement a retry logic for 3 times if the Pull / Push fails. As per the documentation I have created a custom Service Handler(IMobileServiceSyncHandler).
Please find my code logic here.
public class CustomSyncHandler : IMobileServiceSyncHandler
{
public async Task<JObject> ExecuteTableOperationAsync(IMobileServiceTableOperation operation)
{
MobileServiceInvalidOperationException error = null;
Func<Task<JObject>> tryExecuteAsync = operation.ExecuteAsync;
int retryCount = 3;
for (int i = 0; i < retryCount; i++)
{
try
{
error = null;
var result = await tryExecuteAsync();
return result;
}
catch (MobileServiceConflictException e)
{
error = e;
}
catch (MobileServicePreconditionFailedException e)
{
error = e;
}
catch (MobileServiceInvalidOperationException e)
{
error = e;
}
catch (Exception e)
{
throw e;
}
if (error != null)
{
if(retryCount <=3) continue;
else
{
//Need to implement
//Update failed, reverting to server's copy.
}
}
}
return null;
}
public Task OnPushCompleteAsync(MobileServicePushCompletionResult result)
{
return Task.FromResult(0);
}
}
But I am not sure how to handle / revert server copy in case all the 3 retry failed.
In the TODO sample they where reverting it based on the
MobileServicePushFailedException. But which is available when we implement IMobileServiceSyncHandler.
More over if we include custom IMobileServiceSyncHandler it wont execute the code after PushAsync / PullAsync. Even the try catch wont fire in case any exception.
try
{
await this.client.SyncContext.PushAsync();
await this.todoTable.PullAsync(
//The first parameter is a query name that is used internally by the client SDK to implement incremental sync.
//Use a different query name for each unique query in your program
"allTodoItems",
this.todoTable.CreateQuery());
}
catch (MobileServicePushFailedException exc)
{
if (exc.PushResult != null)
{
syncErrors = exc.PushResult.Errors;
}
}
// Simple error/conflict handling. A real application would handle the various errors like network conditions,
// server conflicts and others via the IMobileServiceSyncHandler.
if (syncErrors != null)
{
foreach (var error in syncErrors)
{
if (error.OperationKind == MobileServiceTableOperationKind.Update && error.Result != null)
{
//Update failed, reverting to server's copy.
await error.CancelAndUpdateItemAsync(error.Result);
}
else
{
// Discard local change.
await error.CancelAndDiscardItemAsync();
}
Debug.WriteLine(#"Error executing sync operation. Item: {0} ({1}). Operation discarded.", error.TableName, error.Item["id"]);
}
}
}
Note
In my application I am only trying to achieve retry for 3 time in case any server error. I am not looking for to resolve conflicts. Thant is the reason I haven't added the code for the same.
If someone came across similar issues and resolved it please help.
Stez.
You say you aren't trying to resolve conflicts, but you need to resolve them one way or another (without telling the user what's going on, perhaps) by accepting the server version of the object or updating the client operation. Otherwise it will just keep telling you about the same conflict each time it retries the operation.
You need to have a subclass of the Microsoft.WindowsAzure.MobileServices.Sync.MobileServiceSyncHandler class, which overrides OnPushCompleteAsync() in order to handle conflicts and other errors. Let's call the class SyncHandler:
public class SyncHandler : MobileServiceSyncHandler
{
public override async Task OnPushCompleteAsync(MobileServicePushCompletionResult result)
{
foreach (var error in result.Errors)
{
await ResolveConflictAsync(error);
}
await base.OnPushCompleteAsync(result);
}
private static async Task ResolveConflictAsync(MobileServiceTableOperationError error)
{
Debug.WriteLine($"Resolve Conflict for Item: {error.Item} vs serverItem: {error.Result}");
var serverItem = error.Result;
var localItem = error.Item;
if (Equals(serverItem, localItem))
{
// Items are the same, so ignore the conflict
await error.CancelAndUpdateItemAsync(serverItem);
}
else // check server item and local item or the error for criteria you care about
{
// Cancels the table operation and discards the local instance of the item.
await error.CancelAndDiscardItemAsync();
}
}
}
Include an instance of this SyncHandler() when you initialize your MobileServiceClient:
await MobileServiceClient.SyncContext.InitializeAsync(store, new SyncHandler()).ConfigureAwait(false);
Read up on the MobileServiceTableOperationError to see other conflicts you can handle as well as its methods to allow resolving them.
The exception carries with it a copy of the server version. In my implementation of IMobileServiceSyncHandler I therefore just return error.Value and this seems to work.
A more extensive example of this kind of logic can be found in this MSDN blog.
The same author has another example where he shows how you can resolve the conflict in favour of the server copy or the client copy, here.
Both myself and a colleague have been tasked with finding connection-retry logic for Azure Table Storage. After some searching, I found this really cool Enterprise Library suite, which contains the Microsoft.Practices.TransientFaultHandling namespace.
Following a few code examples, I ended up creating an Incremental retry strategy, and wrapping one of our storage calls with the retryPolicy's ExecuteAction callback handler :
/// <inheritdoc />
public void SaveSetting(int userId, string bookId, string settingId, string itemId, JObject value)
{
// Define your retry strategy: retry 5 times, starting 1 second apart, adding 2 seconds to the interval each retry.
var retryStrategy = new Incremental(5, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(2));
var storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting(StorageConnectionStringName));
try
{
retryPolicy.ExecuteAction(() =>
{
var tableClient = storageAccount.CreateCloudTableClient();
var table = tableClient.GetTableReference(SettingsTableName);
table.CreateIfNotExists();
var entity = new Models.Azure.Setting
{
PartitionKey = GetPartitionKey(userId, bookId),
RowKey = GetRowKey(settingId, itemId),
UserId = userId,
BookId = bookId.ToLowerInvariant(),
SettingId = settingId.ToLowerInvariant(),
ItemId = itemId.ToLowerInvariant(),
Value = value.ToString(Formatting.None)
};
table.Execute(TableOperation.InsertOrReplace(entity));
});
}
catch (StorageException exception)
{
ExceptionHelpers.CheckForPropertyValueTooLargeMessage(exception);
throw;
}
}
}
Feeling awesome, I went to go show my colleague, and he smugly noted that we could do the same thing without having to include Enterprise Library, as the CloudTableClient object already has a setter for a retry policy. His code ended up looking like :
/// <inheritdoc />
public void SaveSetting(int userId, string bookId, string settingId, string itemId, JObject value)
{
var storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting(StorageConnectionStringName));
var tableClient = storageAccount.CreateCloudTableClient();
// set retry for the connection
tableClient.RetryPolicy = new ExponentialRetry(TimeSpan.FromSeconds(2), 3);
var table = tableClient.GetTableReference(SettingsTableName);
table.CreateIfNotExists();
var entity = new Models.Azure.Setting
{
PartitionKey = GetPartitionKey(userId, bookId),
RowKey = GetRowKey(settingId, itemId),
UserId = userId,
BookId = bookId.ToLowerInvariant(),
SettingId = settingId.ToLowerInvariant(),
ItemId = itemId.ToLowerInvariant(),
Value = value.ToString(Formatting.None)
};
try
{
table.Execute(TableOperation.InsertOrReplace(entity));
}
catch (StorageException exception)
{
ExceptionHelpers.CheckForPropertyValueTooLargeMessage(exception);
throw;
}
}
My Question :
Is there any major difference between these two approaches, aside from their implementations? They both seem to accomplish the same goal, but are there cases where it's better to use one over the other?
Functionally speaking both are the same - they both retries requests in case of transient errors. However there are few differences:
Retry policy handling in storage client library only handles retries for storage operations while transient fault handling retries not only handles storage operations but also retries SQL Azure, Service Bus and Cache operations in case of transient errors. So if you have a project where you're using more that storage but would like to have just one approach for handling transient errors, you may want to use transient fault handling application block.
One thing I liked about transient fault handling block is that you can intercept retry operations which you can't do with retry policy. For example, look at the code below:
var retryManager = EnterpriseLibraryContainer.Current.GetInstance<RetryManager>();
var retryPolicy = retryManager.GetRetryPolicy<StorageTransientErrorDetectionStrategy>(ConfigurationHelper.ReadFromServiceConfigFile(Constants.DefaultRetryStrategyForTableStorageOperationsKey));
retryPolicy.Retrying += (sender, args) =>
{
// Log details of the retry.
var message = string.Format(CultureInfo.InvariantCulture, TableOperationRetryTraceFormat, "TableStorageHelper::CreateTableIfNotExist", storageAccount.Credentials.AccountName,
tableName, args.CurrentRetryCount, args.Delay);
TraceHelper.TraceError(message, args.LastException);
};
try
{
var isTableCreated = retryPolicy.ExecuteAction(() =>
{
var table = storageAccount.CreateCloudTableClient().GetTableReference(tableName);
return table.CreateIfNotExists(requestOptions, operationContext);
});
return isTableCreated;
}
catch (Exception)
{
throw;
}
In the code example above, I could intercept retry operations and do something there if I want to. This is not possible with storage client library.
Having said all of this, it is generally recommended to go with storage client library retry policy for retrying storage operations as it is an integral part of the package and thus would be kept up to date with the latest changes to the library.
I had some confusion which I want to clear it - I am inserting values into database using ADO.NET. Let say I want to insert 10 item if I encounter error while inserting data of 5th item it should roll back whatever I had inserted into the database.
I just read the concept of Transaction and Rollback method and also tried to implement it in the program but still it insert 4 item and give me error message of 5th item. It doesn't roll back insert query.
Does transaction and roll back method solved my issue or I need to used other alternative.
here is my code,
for (int i = 0; i < itemLength - 1; i++)
{
//--- Start local transaction ---
myTrans = Class1.conn.BeginTransaction();
//--- Assign transaction object and connection to command object for a pending local transaction ---
_insertQry = Class1.conn.CreateCommand();
_insertQry.Connection = Class1.conn;
_insertQry.Transaction = myTrans;
_insertQry.CommandText = "INSERT INTO Product_PropertyValue(ItemNo, PropertyNo, ValueNo) VALUES (#ItemNo, #PropertyNo, #ValueNo)";
//_insertQry = new SqlCommand("INSERT INTO Product_PropertyValue(ItemNo, PropertyNo, ValueNo) VALUES (#ItemNo, #PropertyNo, #ValueNo)", Class1.conn);
_insertQry.Parameters.AddWithValue("#ItemNo", _itemNo[i]);
_insertQry.Parameters.AddWithValue("#PropertyNo", _propNo);
_insertQry.Parameters.AddWithValue("#ValueNo", _propValue);
_insertQry.ExecuteNonQuery();
myTrans.Commit();
}
Can anyone help me?
It sounds like you are trying to achieve an atomic commit. It either inserts completely or doesn't insert at all.
Try something like the following
SqlTransaction objTrans = null;
using (SqlConnection objConn = new SqlConnection(strConnString))
{
objConn.Open();
objTrans = objConn.BeginTransaction();
SqlCommand objCmd1 = new SqlCommand("insert into tbExample values(1)", objConn);
SqlCommand objCmd2 = new SqlCommand("insert into tbExample values(2)", objConn);
try
{
objCmd1.ExecuteNonQuery();
objCmd2.ExecuteNonQuery();
objTrans.Commit();
}
catch (Exception)
{
objTrans.Rollback();
}
finally
{
objConn.Close();
}
Also take a look at
http://www.codeproject.com/Articles/10223/Using-Transactions-in-ADO-NET
I did 2 modification to your code
1) Move the BeginTransaction() outside the for loop, So that all your 10 INSERt statements are in a single transaction, that is what you want if you want them to be atomic
2) added a TRY/CATCH block, so that you can roll back in case of errors.
//--- Start local transaction ---
myTrans = Class1.conn.BeginTransaction();
bool success = true;
try
{
for (int i = 0; i < itemLength - 1; i++)
{
//--- Assign transaction object and connection to command object for a pending local transaction ---
_insertQry = Class1.conn.CreateCommand();
_insertQry.Connection = Class1.conn;
_insertQry.Transaction = myTrans;
_insertQry.CommandText = "INSERT INTO Product_PropertyValue(ItemNo, PropertyNo, ValueNo) VALUES (#ItemNo, #PropertyNo, #ValueNo)";
//_insertQry = new SqlCommand("INSERT INTO Product_PropertyValue(ItemNo, PropertyNo, ValueNo) VALUES (#ItemNo, #PropertyNo, #ValueNo)", Class1.conn);
_insertQry.Parameters.AddWithValue("#ItemNo", _itemNo[i]);
_insertQry.Parameters.AddWithValue("#PropertyNo", _propNo);
_insertQry.Parameters.AddWithValue("#ValueNo", _propValue);
_insertQry.ExecuteNonQuery();
}
}
catch (Exception ex)
{
success = false;
myTrans.Rollback();
}
if (success)
{
myTrans.Commit();
}
let me know if this doesn't works.
You are on the right path, ADO.NET supports transactions so you will be able to rollback on errors.
Posting your your code here would get you more specific guidance; However since your question is very generic, I will encourage you to follow the template provided by MSDN
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
// Start a local transaction.
SqlTransaction sqlTran = connection.BeginTransaction();
// Enlist a command in the current transaction.
SqlCommand command = connection.CreateCommand();
command.Transaction = sqlTran;
try
{
// Execute two separate commands.
command.CommandText =
"INSERT INTO Production.ScrapReason(Name) VALUES('Wrong size')";
command.ExecuteNonQuery();
command.CommandText =
"INSERT INTO Production.ScrapReason(Name) VALUES('Wrong color')";
command.ExecuteNonQuery();
// Commit the transaction.
sqlTran.Commit();
Console.WriteLine("Both records were written to database.");
}
catch (Exception ex)
{
// Handle the exception if the transaction fails to commit.
Console.WriteLine(ex.Message);
try
{
// Attempt to roll back the transaction.
sqlTran.Rollback();
}
catch (Exception exRollback)
{
// Throws an InvalidOperationException if the connection
// is closed or the transaction has already been rolled
// back on the server.
Console.WriteLine(exRollback.Message);
}
}
}
I found some exceptions from cassandra when I do batch mutation, it said "already has modifications in this mutation", but the info given are two different operations.
I use Super column with counters in this case, it's like
Key: md5 of urls, utf-8
SuperColumnName: date, utf-8
ColumnName: Counter name is a random number from 1 to 200,
ColumnValue:1L
L
public void SuperCounterMutation(ArrayList<String> urlList) {
LinkedList<HCounterSuperColumn<String, String>> counterSuperColumns;
for(String line : urlList) {
String[] ele = StringUtils.split(StringUtils.strip(line), ':');
String key = ele[0];
String SuperColumnName = ele[1];
LinkedList<HCounterColumn<String>> ColumnList = new LinkedList<HCounterColumn<String>>();
for(int i = 2; i < ele.length; ++i) {
ColumnList.add(HFactory.createCounterColumn(ele[i], 1L, ser));
}
mutator.addCounter(key, ColumnFamilyName, HFactory.createCounterSuperColumn(SuperColumnName, ColumnList, ser, ser));
++count;
if(count >= BUF_MAX_NUM) {
try {
mutator.execute();
} catch(Exception e) {
e.printStackTrace();
}
mutator = HFactory.createMutator(keyspace, ser);
count = 0;
}
}
return;
}
Error info from cassandra log showed that the duplicated operations have the same key only, SuperColumnName are not the same, and for counter name set, some conflicts have intersects and some not.
I'm using Cassandra 0.8.1 with hector 0.8.0-rc2
Can anyone tell me the reason of this problem? Thanks in advance!
Error info from cassandra log showed that the duplicated operations have the same key
Bingo. You'll need to combine operations from the same key into a single mutation.