Transaction not getting completed after commit in Azure SQL Data Warehouse - azure

I am trying out transactions using JDBC in Azure SQL Data Warehouse. The transaction is successfully processed, but after the transaction, DDL command fails with error Operation cannot be performed within a transaction.
Here is the what I am trying to do.
connection.createStatement().execute("CREATE TABLE " + schema + ".transaction_table (id INT)");
connection.createStatement().execute("INSERT INTO " + schema + ".transaction_table (id) VALUES (1)");
connection.createStatement().execute("INSERT INTO " + schema + ".transaction_table (id) VALUES (2)");
// Transaction starts
connection.setAutoCommit(false);
connection.createStatement().execute("DELETE FROM " + schema + ".transaction_table WHERE id = 2");
connection.createStatement().execute("INSERT INTO " + schema + ".transaction_table (id) VALUES (10)");
connection.commit();
connection.setAutoCommit(true);
// Transaction ends
// Next DDL command to succeed, but it does not
connectiom.createStatement().execute("CREATE TABLE " + schema + ".transaction_table_new (id INT)");
// Fails with `Operation cannot be performed within a transaction`
So, how can we close the transaction in Azure SQL Data Warehouse.
I tried to do it like this.
try {
// This fails
connection.createStatement().execute("CREATE TABLE " + schema + ".transaction_table_new (id INT)");
} catch (SQLServerException e) {
if (e.getMessage().contains("Operation cannot be performed within a transaction")) {
// This succeeds
// Somehow the transaction was closed, may be because of the exception
connection.createStatement().execute("CREATE TABLE " + schema + ".transaction_table_new "(id INT)");
}
}

SQL Data Warehouse expects the CREATE TABLE statement to be run outside of a transaction. By setting the connection.setAutoCommit to true, you are forcing Java to run the execute within a transaction. I'm a bit weak on Java (it's been a while) but you should be able to run the second DDL statement by simply commenting out the setAutoCommit(true) line. This will leave the JDBC driver in an execute mode only and not run the execute() operation within a transaction.

It looks like we have to end the transaction manually.
It looks like this
connection.setAutoCommit(false);
// Transaction statement 1
// Transaction statement 2
connection.commit();
connection.setAutoCommit(true);
connection.createStatement().execute("IF ##TRANCOUNT > 0 COMMIT TRAN");
This is because, for Azure SQL Data Warehouse, jdbc connection.commit() doesn’t appear to always issue the COMMIT. It keeps track of transactions it’s managing and decides to be “smart” about what it sends. So manual COMMIT TRAN is executed to close all the open transactions before executing any DDL commands.
This is strange as we don't have to do this for other warehouses or databases, but it works. And, this is not documented.

Related

Spring Integration: Aggregator to expire message on timeout

I am using SI's aggregator pattern to hold events and wait for the completion events and storing it in JdbcMessage store. I have created the table INT_MESSAGE, INT_MESSAGE_GROUP and INT_GROUP_TO_MESSAGE.
Sometimes, the completion event may not be available and I want to complete and discard the event, remove it from the tables. I don't want the tables to grow big un-necessarily
I have specified the below config in the pipeline
.expireGroupsUponCompletion(true)
.expireGroupsUponTimeout(true)
.groupTimeout(groupMessageTimeOut)
.sendPartialResultOnExpiry(false)
Would this ensure if the completion event doesn't arrive in x minutes then the message group will be expired, discarded in the null channel and removed from the tables.
Please suggest.
Your summary is correct. Both .expireGroupsUponCompletion(true) & .expireGroupsUponTimeout(true) do remove a group from the store.
The sendPartialResultOnExpiry(false) really does what you are asking:
if (this.sendPartialResultOnExpiry) {
if (this.logger.isDebugEnabled()) {
this.logger.debug("Prematurely releasing partially complete group with key ["
+ correlationKey + "] to: " + getOutputChannel());
}
completeGroup(correlationKey, group, lock);
}
else {
if (this.logger.isDebugEnabled()) {
this.logger.debug("Discarding messages of partially complete group with key ["
+ correlationKey + "] to: "
+ (this.discardChannelName != null ? this.discardChannelName : this.discardChannel));
}
if (this.releaseLockBeforeSend) {
lock.unlock();
}
group.getMessages()
.forEach(this::discardMessage);
}
Tell us, please, what made you to be confused about that configuration?

Cosmos DB: Gremlin API Request too large exception. How to retry the call

I have a Throughput of 1000 RU/s in my Azure Cosmos DB and I have around 290 queries to be executed. I keep getting request too large exception.
Each query have 12 properties and 1 Partition key but I still think with 1000 RU/s the queries should be executed properly.
I have a gremlinClient
public static GremlinClient GetGremlinClient()
{
var gremlinServer = new GremlinServer(Endpoint, Port, enableSsl: true,
username: "/dbs/" + Databasename + "/colls/" + Collectionname, password: Authkey);
var gremlinClient = new GremlinClient(gremlinServer, new GraphSON2Reader(), new GraphSON2Writer(),
GremlinClient.GraphSON2MimeType);
return gremlinClient;
}
A sample query. I am just trying to add vertices
g.addV('Experience').property('_test', 'dummy').property('someProperty', 'dummy').property('someProperty', 'dummy').property('someProperty', 'Documentation of the business processes
of all departments as well as the management level for an informed
selection of an ERP-system for a medium-sized industrial enterprise;
Role: Project management ').property('someProperty',
'2016').property('someProperty', 'Offen').property('someProperty',
'Dummy').property('someProperty', 'EN').property('someProperty',
'Industry').property('someProperty', 'Process documentation of
the whole company for a profounded selection of an ERP-System.')
That for-each executes all the queries
foreach (string query in queries)
{
await gremlinClient.SubmitAsync<dynamic>(query);
}
The error I get
Server error: \r\n\nActivityId : 2312f64f-b865-49cc-bb26-843d46313199\nExceptionType : RequestRateTooLargeException\nExceptionMessage :\r\n\tMessage: {\"Errors\":[\"Request rate is large\"]}\r\n\tActivityId: 157daf87-3238-4e1c-9a81-41bcd6d7c2e1, Request URI: /apps/413f848b-ce17-40fc-ad7f-14c0e21e9633/services/29abd22a-4e74-48c1-aab3-b311be968829/partitions/9e4cb405-4f74-4d7f-8d12-26e79b910143/replicas/132142016542682221s/, RequestStats: \r\n\tRequestStartTime: 2019-10-24T09:27:38.2395067Z, RequestEndTime: 2019-10-24T09:27:38.2395067Z, Number of regions attempted:1\r\n\tResponseTime: 2019-10-24T09:27:38.2395067Z
Its a simple code I dont understand what I can change in that.
Is there a way to retry the request for the same point or somehow not get the error or to avoid the error
The exception you receive is Request*Rate*TooLargeException, meaning you are submitting too many request in a short period of time.
For running bulk operations you should use the vendor specific tooling.

Microsoft.Azure.Cosmos.Table - How could I retrieve an item operation status if it is inserted or merged using insertOrMergeOperation?

I'm recently using the Microsoft.Azure.Cosmos.Table API and I noticed there was a great method called InsertOrMergeOperation, however, I am wondering if there is any way that the return result could tell me whether my entity just did an insert or a merge/update operation. I checked the TableResult object but it doesn't return any useful information about that.
Is there anyone know if this Operation could give me what I want? If not, is there any other operation that could perform the same workflow?
The TableResule does not return a value to indicate if it's an insert or merge operation. If you want to get the info, you have 2 methods for that:
Method 1:use table query with partition key and rowkey to check if the records exists or not in table, then you know the following operation by InsertOrMerge() is insert or merge as per the result is null or not, sample code like below:
CloudTableClient tableClient = account.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("People");
TableQuery<CustomerEntity> query = new TableQuery<CustomerEntity>()
.Where(
TableQuery.CombineFilters(
TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, "Smith"),
TableOperators.And,
TableQuery.GenerateFilterCondition("Email", QueryComparisons.Equal,"Ben#contoso.com")
));
await table.ExecuteQuerySegmentedAsync<CustomerEntity>(query, null);
Method 2:use try-catch block, and first do a insert operation, if it returns an "conflict" error message, means the following InsertOrMerge() operation is an merge operation, code like below:
TableOperation t1;
try
{
t1 = TableOperation.Insert(customer);
table.Execute(t1);
}
catch (Exception e)
{
if (e.Message.ToLower() == "Conflict".ToLower())
{
t1 = TableOperation.InsertOrMerge(customer);
table.Execute(t1);
}
}
Hope it helps.

Cassandra Trigger Exception: InvalidQueryException: table of additional mutation does not match primary update table

i am using Cassandra Trigger on a table. I am following the example and loading trigger jar with 'nodetool reloadtriggers'. Then i am using
'CREATE TRIGGER mytrigger ON ..'
command from cqlsh to create trigger on my table.
Adding an entry into that table , my audit table is being populated.
But calling a method from within my Java application, which persists an entry into my table by using
'session.execute(BoundStatement)' i am getting this exception:
InvalidQueryException: table of additional mutation does not match primary update table
Why does the insertion into the table and the audit work when doing it directly with cqlsh and why does it fail when doing pretty much exactly the same with the Java application?
i am using this as AuditTrigger, very simplified(left out all of the other operations other than Row insertion:
public class AuditTrigger implements ITrigger {
private Properties properties = loadProperties();
public Collection<Mutation> augment(Partition update) {
String auditKeyspace = properties.getProperty("keyspace");
String auditTable = properties.getProperty("table");
CFMetaData metadata = Schema.instance.getCFMetaData(auditKeyspace,
auditTable);
PartitionUpdate.SimpleBuilder audit =
PartitionUpdate.simpleBuilder(metadata, UUIDGen.getTimeUUID());
if (row.primaryKeyLivenessInfo().timestamp() != Long.MIN_VALUE) {
// Row Insertion
JSONObject obj = new JSONObject();
obj.put("message_id", update.metadata().getKeyValidator()
.getString(update.partitionKey().getKey()));
audit.row().add("operation", "ROW INSERTION");
}
audit.row().add("keyspace_name", update.metadata().ksName)
.add("table_name", update.metadata().cfName)
.add("primary_key", update.metadata().getKeyValidator()
.getString(update.partitionKey()
.getKey()));
return Collections.singletonList(audit.buildAsMutation());
It seems like using BoundStatement, the trigger fails:
session.execute(boundStatement);
, using a regular cql queryString works though.
session.execute(query)
We are using Boundstatement everywhere within our application though and cannot change that.
Any help would be appreciated.
Thanks

Lagom framework / Persistent Read Side / Cassandra / DataStax / Table unconfigured

I successfully compiled the code example from http://www.lagomframework.com/documentation/1.0.x/ReadSide.html
It's about the read-side of the CQRS schema.
There is only problem: it doesn't run.
Looks like configuration problem... and the official documentation of Lagom at this point is very incomplete.
The error says:
java.util.concurrent.CompletionException: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured table postsummary
Alright, there's a line in the code that does cassandra query, selecting & inserting from & to a table named postsummary.
I thought the tables are auto-created by default. Anyway, in doubt, I simply added this line to my application.conf:
cassandra-journal.keyspace-autocreate = true
cassandra-journal.tables-autocreate = true
Still..., no luck, same error after restarting.
Maybe it has something to do with another error during startup, that says:
[warn] a.p.c.j.CassandraJournal - Failed to connect to Cassandra and initialize. It will be retried on demand. Caused by: ServiceLocator is not bound
I thought... alright, maybe it's trying to contact 9042 (default cassandra port), while lagom by default starts embedded cassandra at 4000.
So I tried adding these lines in application.conf:
cassandra-journal.contact-points = ["127.0.0.1"]
cassandra-journal.port = 4000
lagom.persistence.read-side.cassandra.contact-points = ["127.0.0.1"]
lagom.persistence.read-side.cassandra.port = 4000
Still..., no luck, same error.
Can anyone help me solve it. I need to get this example running, crucial part of CQRS study using lagom.
Some ref.: https://github.com/lagom/lagom/blob/master/persistence/src/main/resources/reference.conf
Here are some screenshots:
Btw, I solved it by creating the tables inside the code, calling this method from the prepare method of the event processor:
private CompletionStage<Done> prepareTables(CassandraSession session) {
CompletionStage<Done> preparePostSummary = session.executeCreateTable(
"CREATE TABLE IF NOT EXISTS postsummary ("
+ "partition bigint, id text, title text, "
+ "PRIMARY KEY (id))"
).whenComplete((ok, err) -> {
if (err != null) {
System.out.println("Failed to create postsummary table, due to: " + err.getMessage());
}
});
CompletionStage<Done> prepareBlogEventOffset = session.executeCreateTable(
"CREATE TABLE IF NOT EXISTS blogevent_offset ("
+ "partition bigint, offset uuid, "
+ "PRIMARY KEY (offset))"
).whenComplete((ok, err) -> {
if (err != null) {
System.out.println("Failed to create blogevent_offset table, due to: " + err.getMessage());
}
});
return preparePostSummary.thenCompose(a -> prepareBlogEventOffset);
}
Thanks!,
Raka
I have a working example here. Even if it does not use auto created tables :
https://github.com/lagom/activator-lagom-cargotracker/blob/master/registration-impl/src/main/java/sample/cargotracker/registration/impl/CargoEventProcessor.java

Resources