I set up my nodejs app with qldb to implement a wallet service. Set up some tests with some success tests and some expected error tests and once in a while this error 'BadRequestException: no open transaction' would happen and cause my tests to fail. if I run the test again, they will pass. Again once in a while, this error will happen unexpectedly cause the tests to fail. I noted when commented out my expected error tests and the error didn't happen or did not happen as often. and this error happens not only to the expected error test but to the successful tests.
this is how my tests look like
describe('createWallet()', () => {
it('should return an object with wallet Id', async () => {
let result6 = await controller.createWallet({ body: mocks.walletInfo6.info });
documentId6 = result6.walletId;
expect(result6).to.have.property('walletId').that.is.a.uuid;
});
it('One player should have only one active wallet for each currency', async () => {
try {
let res = await controller.createWallet({ body: mocks.walletInfo1.info });
assert.fail('expected error was not thrown')
} catch (e) {
expect(e.message).to.equal('Player already owns an active wallet in this currency.');
}
});
});
describe('suspendWallet()', () => {
it('should change wallet status to suspend', async () => {
let res = await controller.suspendWallet({ documentId: documentId3 });
await controller.suspendWallet({ documentId: documentId5 });
expect(res).to.be.a.string;
expect(res).to.equal(documentId3);
});
it('should not change wallet status if wallet Id is invalid', async () => {
try {
let res = await controller.suspendWallet({ documentId: mocks.invalidWalletId });
assert.fail('expected error was not thrown')
} catch (e) {
expect(e.message).to.equal('Did not find any record with this document Id.');
}
});
});
It's hard to be certain about how your application is running into this error without looking at how the driver is being used to execute transactions.
The driver APIs (for example - execute ) returns a promise. One way the application could be seeing the "No transaction open error" is that the promise is not resolved before sending further commands.
Cookbook - Refer to the QLDB JS driver cookbook which lists code sample for CRUD operations. Note how the samples use await inside the transactions to wait for the promises to resolve. Not waiting for the promises returned by execute can cause the driver to commit the transaction before the execute call is processed and hence a "No open transaction error".
Sample code for executing transactions -
var qldb = require('amazon-qldb-driver-nodejs');
const qldbDriver = new qldb.QldbDriver("vehicle-registration");
(async function() {
await qldbDriver.executeLambda(async (txn) => {
await txn.execute("CREATE TABLE Person");
});
})();
In case you still face issues, please share the code snippet where you use the driver to execute transactions.
Update on this issue. I use nodejs driver version 2.1.0.
My team and I found out that the problem was because there are rollbacks that happen after the error tests and we don't know when the rollbacks are done. When the rollback of the previous test is still running, the transaction for that test is still open so if the next test tries to open a new transaction it would conflict and would not able to open a new transaction for the next test. To fix this, we just not throw errors inside the transaction to prevent rollbacks to happen. This way works for our code, but a better solution would be a way to detect when the rollback is done from the driver and wait for the transaction to close before opening a new transaction.
Related
I often have situations where the behavior I'm trying to achieve is like this:
User takes action
Website updates UI optimistically
Website fires update to server
Website awaits server response
If the update fails, website rolls back UI change
I've often found myself adding multiple timeouts to my tests to try to both assert that the optimistic update was made and then gets rolled back after rejection. So something like this:
it('rolls back optimistic update', async () => {
jest.mocked(doUpdate).mockReturnValue(new Promise((resolve, reject) => {
setTimeout(reject, 1000);
});
render(<App />)
await screen.findByText('not done')
userEvent.click(screen.getByText('do it'))
await screen.findByText('Done!')
await screen.findByText('not done')
});
But this has some pretty big downsides:
Setting up tests like this is fidgety and difficult.
Using timeouts inside my tests results in tests that are ensured to be slower than they need to be.
Any significant changes to the test environment or tools have a high chance of breaking these tests.
Tests get even slower and more complicated once I need to test things like ensuring that subsequent user actions don't clobber previous actions.
If the promise happens to resolve after the test ends, I often end up with React complaining about updates not being wrapped in act.
How can I test this type of behavior in a way that is as efficient as possible and robust against test environment changes?
I now use this helper function in situations like this, where I want to control the precise order of events and promises are involved:
export default function loadControlledPromise(mock: any) {
let resolve: (value?: unknown) => void = () => {
throw new Error('called resolve before definition');
};
let reject: (value?: unknown) => void = () => {
throw new Error('called reject before definition');
};
const response = new Promise((res, rej) => {
resolve = res;
reject = rej;
});
jest.mocked(mock).mockReturnValue(response);
return { resolve, reject };
}
Using this helper function, the example test becomes:
it('rolls back optimistic update', async () => {
const { reject } = loadControlledPromise(doUpdate);
render(<App />)
await screen.findByText('not done')
userEvent.click(screen.getByText('do it'))
await screen.findByText('Done!')
reject();
await screen.findByText('not done')
});
This ensures:
The test doesn't spend more time waiting than necessary.
Changes in the test environment are less likely to break the test.
More-complicated sequences of events can be tested without resulting in incomprehensible test code.
I can force promises to resolve before the test ends. even when I need to test conditions prior to the promise resolving, helping to avoid React complaining about updates happening outside of act.
I am working on setting up an automation test suite for an application using selenium and jest and it turns out that the console never reaches the inner body of the minimalist test case written below.
describe('When a user Opens Launchpad', () => {
test('It should be able to Navigate to Tasks Application without errors', async () => {
driver.get('http://localhost:4004/fiori.html').then(function () {
const temp = driver.findElement(By.xpath("li[#id='__tile11']"));
temp.then(function (element){
element.getAttribute("innerHTML");
expect(element.getText()).toBe("VT Dashboard");
})
});
}, 200000);
});
I looked online and tried multiple fixes like putting the driver.get() method above all these functions, making the test cases synchronous and using getText() instead of getAttribute() but none of them worked.
I either get an element not found error (The element actually exists when I check it on the chromium browser) or the test case executes successfully without reaching the expect statement in debug mode.
Bottomline is that driver.findElement() returns a promise instead of an element and would be great if I could get an element instead of promise.
Any help or correction here would be greatly appreciated.
If the function is async you should return the promise chain from that function or just use await keyword. So try following:
test('It should be able to Navigate to Tasks Application without errors', async () => {
await driver.get('http://localhost:4004/fiori.html');
const temp = await driver.findElement(By.xpath("li[#id='__tile11']"));
element.getAttribute("innerHTML");
expect(element.getText()).toBe("VT Dashboard");
}, 200000);
or
test('It should be able to Navigate to Tasks Application without errors', async () => {
return driver.get('http://localhost:4004/fiori.html').then(function () {
const temp = driver.findElement(By.xpath("li[#id='__tile11']"));
temp.then(function (element){
element.getAttribute("innerHTML");
expect(element.getText()).toBe("VT Dashboard");
})
});
}, 200000);
I am using Testcafe for E2E automation and I want to add some prettier errors (instead of x element not found in DOM). Before each test, I do a clean login to the application, but that might fail (since I run the tests in DEV environment after a successful master build deployment), such as, devs broke the build, a service essential for login is down, the login itself broke and so on.
I want my script to fail right away instead of trying to run all tests and fail with the same reason. I read on stack overflow that I should use a return and that my script should stop with return, that's why I added it in the second catch block (retry login, then fail), but it doesn't seem to do that. What am I doing wrong?
Should I use a try/catch with a return in the loginpage.login or I can achieve what I want to achieve in the before each hook?
Right now, if login fails for whatever reason, the script still attempts to execute all tests. I want the tests to be skipped if the before each hook failed. How can I achieve this?
fixture `Activity History Tests`
.page `https://mypage.com`
.beforeEach(async (t) => {
try {
await loginPage.login(username, password)
}
catch(e) {
try {
console.log('Login Failed, retrying. Failure reason: ')
await loginPage.login(username, password)
}
catch(e) {
throw new Error('Couldn\'t login')
return
}
}}
)
test('Test #1: do something, async (t) => {
await t.setTestSpeed(0.8)
try { //the test goes here }
The contents of loginPage.login() are:
async login(username: string, password: string) {
await t
.setTestSpeed(0.5)
.maximizeWindow()
.click(this.loginButton)
await t
.typeText(this.emailField, username, {paste:true})
.click(this.submit)
await this.passwordField.visible
await t
.typeText(this.passwordField, password)
.click(this.signInButton)
// await t
//.click(this.staySignedIn)
const breakPanelText = await this.breakPanel
await t
.expect(breakPanelText.innerText).notContains("Hello Agent", {timeout: 10000})
};
At present, there is no public API to abort test execution using the Test API. I've created a suggestion in the TestCafe GitHub repository for your use case.
When working with a big application that has several tables and several DB operations it's very difficult to keep track of what transactions are occurring. To workaround this we started by passing around a trx object.
This has proven to be very messy.
For example:
async getOrderById(id: string, trx?: Knex.Transaction) { ... }
Depending on the function calling getOrderById it will either pass a trx object or not. The above function will use trx if it is not null.
This seems simple at first, but it leads to mistakes where if you're in the middle of a transaction in one function and call another function that does NOT use a transaction, knex will hang with famous Knex: Timeout acquiring a connection. The pool is probably full.
async getAllPurchasesForUser(userId: string) {
..
const trx = await knex.transaction();
try {
..
getPurchaseForUserId(userId); // Forgot to make this consume trx, hence Knex timesout acquiring connection.
..
}
Based on that, I'm assuming this is not a best practice, but I would love if someone from Knex developer team could comment.
To improve this we're considering to instead use knex.transactionProvider() that is accessed throughout the app wherever we perform DB operations.
The example on the website seems incomplete:
// Does not start a transaction yet
const trxProvider = knex.transactionProvider();
const books = [
{title: 'Canterbury Tales'},
{title: 'Moby Dick'},
{title: 'Hamlet'}
];
// Starts a transaction
const trx = await trxProvider();
const ids = await trx('catalogues')
.insert({name: 'Old Books'}, 'id')
books.forEach((book) => book.catalogue_id = ids[0]);
await trx('books').insert(books);
// Reuses same transaction
const sameTrx = await trxProvider();
const ids2 = await sameTrx('catalogues')
.insert({name: 'New Books'}, 'id')
books.forEach((book) => book.catalogue_id = ids2[0]);
await sameTrx('books').insert(books);
In practice here's how I'm thinking about using this:
SingletonDBClass.ts:
const trxProvider = knex.transactionProvider();
export default trxProvider;
Orders.ts
import trx from '../SingletonDBClass';
..
async getOrderById(id: string) {
const trxInst = await trx;
try {
const order = await trxInst<Order>('orders').where({id});
trxInst.commit();
return order;
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to fetch order, error: ${e}`);
}
}
..
Am I understanding this correctly?
Another example function where a transaction is actually needed:
async cancelOrder(id: string) {
const trxInst = await trx;
try {
trxInst('orders').update({ status: 'CANCELED' }).where({ id });
trxInst('active_orders').delete().where({ orderId: id });
trxInst.commit();
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to cancel order, error: ${e}`);
}
}
Can someone confirm if I'm understanding this correctly? And more importantly if this is a good way to do this. Or is there a best practice I'm missing?
Appreciate your help knex team!
No. You cannot have global singleton class returning the transaction for your all of your internal functions. Otherwise you are trying always to use the same transaction for all the concurrent users trying to do different things in the application.
Also when you once commit / rollback the transaction returned by provider, it will not work anymore for other queries. Transaction provider can give you only single transaction.
Transaction provider is useful in a case, where you have for example middleware, which provides transaction for request handlers, but it should not be started, since it might not be needed so you don't want yet allocate a connection for it from pool.
Good way to do your stuff is to pass transcation or some request context or user session around, so that each concurrent user can have their own separate transactions.
for example:
async cancelOrder(trxInst, id: string) {
try {
trxInst('orders').update({ status: 'CANCELED' }).where({ id });
trxInst('active_orders').delete().where({ orderId: id });
trxInst.commit();
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to cancel order, error: ${e}`);
}
}
Depending on the function calling getOrderById it will either pass a trx object or not. The above function will use trx if it is not null.
This seems simple at first, but it leads to mistakes where if you're in the middle of a transaction in one function and call another function that does NOT use a transaction, knex will hang with famous Knex: Timeout acquiring a connection. The pool is probably full.
We usually do it in a way that if trx is null, query throws an error, so that you need to explicitly pass either knex / trx to be able to execute the method and in some methods trx is actually required to be passed.
Anyhow if you really want to force everything to go through single transaction in a session by default you could create API modules in a way that for each user session you create an API instance which is initialized with transaction:
const dbForSession = new DbService(trxProvider);
const users = await dbForSession.allUsers();
and .allUsers() does something like return this.trx('users');
I am using Mongoose to access to my database. I need to use transactions to make an atomic insert-update.
95% of the time my transaction works fine, but 5% of the time an error is showing :
"Given transaction number 1 does not match any in-progress transactions"
It's very difficult to reproduce this error, so I really want to understand where it is coming from to get rid of it.
I could not find a very clear explanation about this type of behaviour.
I have tried to use async/await key words on various functions. I don't know if an operation is not done in time or too soon.
Here the code I am using:
export const createMany = async function (req, res, next) {
if (!isIterable(req.body)) {
res.status(400).send('Wrong format of body')
return
}
if (req.body.length === 0) {
res.status(400).send('The body is well formed (an array) but empty')
return
}
const session = await mongoose.startSession()
session.startTransaction()
try {
const packageBundle = await Package.create(req.body, { session })
const options = []
for (const key in packageBundle) {
if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
options.push({
updateOne: {
filter: { _id: packageBundle[key].id },
update: {
$set: {
custom_id_string: 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
minimumIntegerDigits: 14,
useGrouping: false
})
},
upsert: true
}
}
})
}
}
await Package.bulkWrite(
options,
{ session }
)
for (const key in packageBundle) {
if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
packageBundle[key].custom_id_string = 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
minimumIntegerDigits: 14,
useGrouping: false
})
}
}
res.status(201).json(packageBundle)
await session.commitTransaction()
} catch (error) {
res.status(500).end()
await session.abortTransaction()
throw error
} finally {
session.endSession()
}
}
I expect my code to add in the database and to update the entry packages in atomic way, that there is no instable database status.
This is working perfectly for the main part, but I need to be sure that this bug is not showing anymore.
You should use the session.withTransaction() helper function to perform the transaction, as pointed in mongoose documentation. This will take care of starting, committing and retrying the transaction in case it fails.
const session = await mongoose.startSession();
await session.withTransaction(async () => {
// Your transaction methods
});
Explanation:
The multi-document transactions in MongoDB are relatively new and might be a bit unstable in some cases, such as described here. And certainly, it has also been reported in Mongoose here. Your error most probably is a TransientTransactionError due to a write-conflict happening when the transaction is committed.
However, this is a known and expected issue from MongoDB and these comments explain their reasoning behind why they decided it to be like this. Moreover, they claim that the user should be handling the cases of write conflicts and retrying the transaction if that happens.
Therefore, looking at your code, the Package.create(...) method seems to be the reason why the error gets triggered, since this method is executing a save() for every document in the array (from mongoose docs).
A quick solution might be using Package.insertMany(...) instead of create(), since the Model.insertMany() "only sends one operation to the server, rather than one for each document" (from mongoose docs).
However, MongoDB provides a helper function session.withTransaction() that will take care of starting and committing the transaction and retry it in case of any error, since release v3.2.1. Hence, this should be your preferred way to work with transactions in a safer way; which is, of course, available in Mongoose through the Node.js API.
The accepted answer is great. In my case, I was running multiple transactions serially within a session. I was still facing this issue every now and then. I wrote a small helper to resolve this.
File 1:
// do some work here
await session.withTransaction(() => {});
// ensure the earlier transaction is completed
await ensureTransactionCompletion(session);
// do some more work here
await session.withTransaction(() => {});
Utils File:
async ensureTransactionCompletion(session: ClientSession, maxRetryCount: number = 50) {
// When we are trying to split our operations into multiple transactions
// Sometimes we are getting an error that the earlier transaction is still in progress
// To avoid that, we ensure the earlier transaction has finished
let count = 0;
while (session.inTransaction()) {
if (count >= maxRetryCount) {
break;
}
// Adding a delay so that the transaction get be committed
await new Promise(r => setTimeout(r, 100));
count++;
}
}