I've got a function setClaims which is simply setting claims for current transaction. I need to create a helper function which will check if uuid was provided, set claims and return the transaction object. Something like this:
export const useAuthTransaction = async (
kx: Knex,
setClaims: (tx: Knex.Transaction, uuid?: string) => Promise<any>,
uuid?: string,
): Promise<Knex.Transaction> => {
if (!uuid) {
throw { errorCodes: [RemoveQuestionErrorCode.Unauthorized] };
}
return kx.transaction(async tx => {
await setClaims(tx);
return tx;
});
};
But when I try to use it in my resolver:
useAuthTransaction(kx, setClaims, claims?.uid).then(async tx =>
dataSources.questionAPI.update(tx, input));
it says:
Transaction query already complete
How do I resolve transaction context without closing it?
Returning promise from transaction handler automatically commits / rejects it.
So you are using async tx => {...} type of handler which implicitly always returns a promise.
What you want to do is to use the knex.transactionProvider() feature (check examples from docuementation) and then you can pass that transaction around and tell explicitly when it should be committed / rolled back.
Related
I have a function that returns a Promise and inside that Promise I get an object in the resolve.
Here it is the function of my service that works good.
buscarUsuario(email: string){
return new Promise((resolve, reject) => {
this.http.post(`${URL}/user/email`, {email})
.subscribe(resp => {
//console.log(resp['usuario']);
resolve(resp['usuario']);
});
})
}
And then I get the value from the promise in this var:
const getDatos = this.usuarioService.buscarUsuario(this.correoUsuario.value.toString());
And then I call the var to get the value from the resolve and I can't extract that value from there:
var usuario: Usuario;
getDatos.then(usu => {
usuario = usu;
//Here I can see the value
console.log(usuario);
});
//But here I can't see the value
//And it's where I really need to get the value
console.log(usuario);
So, how do I get that value outside the Promise?
Using Promises in Angular is NOT recommended. Angular recommends use of Observable to handle asynchronous operations
Lets Try and change your code to return Observables only
buscarUsuario = (email: string) =>
this.http.post<any>(`${URL}/user/email`, {email}).pipe(
map(resp => resp.usuario as Usuario)
)
Basically the above code returns an Observable<any>(Observable of type any). I have type casted using <any> to transform the result to an Obserable<any>. Next I have use piping to extract the usuario from response
Now we can actually assign this value to a variable...
const getDatos$ = this.usuarioService.buscarUsuario(this.correoUsuario.value.toString());
NOTE: This is an Observable and you will need to subscribe to it
Observable can be assigned like any other property
const usuario: Observable<Usuario> = getDatos$
You cannot get the value outside that function, reason because since you use promise, you need to wait until the promise returns the value, so best possible way is you have do rest of your functionality within promise.then.
getDatos.then(usu => {
//implement your other functionality
});`
When working with a big application that has several tables and several DB operations it's very difficult to keep track of what transactions are occurring. To workaround this we started by passing around a trx object.
This has proven to be very messy.
For example:
async getOrderById(id: string, trx?: Knex.Transaction) { ... }
Depending on the function calling getOrderById it will either pass a trx object or not. The above function will use trx if it is not null.
This seems simple at first, but it leads to mistakes where if you're in the middle of a transaction in one function and call another function that does NOT use a transaction, knex will hang with famous Knex: Timeout acquiring a connection. The pool is probably full.
async getAllPurchasesForUser(userId: string) {
..
const trx = await knex.transaction();
try {
..
getPurchaseForUserId(userId); // Forgot to make this consume trx, hence Knex timesout acquiring connection.
..
}
Based on that, I'm assuming this is not a best practice, but I would love if someone from Knex developer team could comment.
To improve this we're considering to instead use knex.transactionProvider() that is accessed throughout the app wherever we perform DB operations.
The example on the website seems incomplete:
// Does not start a transaction yet
const trxProvider = knex.transactionProvider();
const books = [
{title: 'Canterbury Tales'},
{title: 'Moby Dick'},
{title: 'Hamlet'}
];
// Starts a transaction
const trx = await trxProvider();
const ids = await trx('catalogues')
.insert({name: 'Old Books'}, 'id')
books.forEach((book) => book.catalogue_id = ids[0]);
await trx('books').insert(books);
// Reuses same transaction
const sameTrx = await trxProvider();
const ids2 = await sameTrx('catalogues')
.insert({name: 'New Books'}, 'id')
books.forEach((book) => book.catalogue_id = ids2[0]);
await sameTrx('books').insert(books);
In practice here's how I'm thinking about using this:
SingletonDBClass.ts:
const trxProvider = knex.transactionProvider();
export default trxProvider;
Orders.ts
import trx from '../SingletonDBClass';
..
async getOrderById(id: string) {
const trxInst = await trx;
try {
const order = await trxInst<Order>('orders').where({id});
trxInst.commit();
return order;
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to fetch order, error: ${e}`);
}
}
..
Am I understanding this correctly?
Another example function where a transaction is actually needed:
async cancelOrder(id: string) {
const trxInst = await trx;
try {
trxInst('orders').update({ status: 'CANCELED' }).where({ id });
trxInst('active_orders').delete().where({ orderId: id });
trxInst.commit();
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to cancel order, error: ${e}`);
}
}
Can someone confirm if I'm understanding this correctly? And more importantly if this is a good way to do this. Or is there a best practice I'm missing?
Appreciate your help knex team!
No. You cannot have global singleton class returning the transaction for your all of your internal functions. Otherwise you are trying always to use the same transaction for all the concurrent users trying to do different things in the application.
Also when you once commit / rollback the transaction returned by provider, it will not work anymore for other queries. Transaction provider can give you only single transaction.
Transaction provider is useful in a case, where you have for example middleware, which provides transaction for request handlers, but it should not be started, since it might not be needed so you don't want yet allocate a connection for it from pool.
Good way to do your stuff is to pass transcation or some request context or user session around, so that each concurrent user can have their own separate transactions.
for example:
async cancelOrder(trxInst, id: string) {
try {
trxInst('orders').update({ status: 'CANCELED' }).where({ id });
trxInst('active_orders').delete().where({ orderId: id });
trxInst.commit();
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to cancel order, error: ${e}`);
}
}
Depending on the function calling getOrderById it will either pass a trx object or not. The above function will use trx if it is not null.
This seems simple at first, but it leads to mistakes where if you're in the middle of a transaction in one function and call another function that does NOT use a transaction, knex will hang with famous Knex: Timeout acquiring a connection. The pool is probably full.
We usually do it in a way that if trx is null, query throws an error, so that you need to explicitly pass either knex / trx to be able to execute the method and in some methods trx is actually required to be passed.
Anyhow if you really want to force everything to go through single transaction in a session by default you could create API modules in a way that for each user session you create an API instance which is initialized with transaction:
const dbForSession = new DbService(trxProvider);
const users = await dbForSession.allUsers();
and .allUsers() does something like return this.trx('users');
I am using Mongoose to access to my database. I need to use transactions to make an atomic insert-update.
95% of the time my transaction works fine, but 5% of the time an error is showing :
"Given transaction number 1 does not match any in-progress transactions"
It's very difficult to reproduce this error, so I really want to understand where it is coming from to get rid of it.
I could not find a very clear explanation about this type of behaviour.
I have tried to use async/await key words on various functions. I don't know if an operation is not done in time or too soon.
Here the code I am using:
export const createMany = async function (req, res, next) {
if (!isIterable(req.body)) {
res.status(400).send('Wrong format of body')
return
}
if (req.body.length === 0) {
res.status(400).send('The body is well formed (an array) but empty')
return
}
const session = await mongoose.startSession()
session.startTransaction()
try {
const packageBundle = await Package.create(req.body, { session })
const options = []
for (const key in packageBundle) {
if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
options.push({
updateOne: {
filter: { _id: packageBundle[key].id },
update: {
$set: {
custom_id_string: 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
minimumIntegerDigits: 14,
useGrouping: false
})
},
upsert: true
}
}
})
}
}
await Package.bulkWrite(
options,
{ session }
)
for (const key in packageBundle) {
if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
packageBundle[key].custom_id_string = 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
minimumIntegerDigits: 14,
useGrouping: false
})
}
}
res.status(201).json(packageBundle)
await session.commitTransaction()
} catch (error) {
res.status(500).end()
await session.abortTransaction()
throw error
} finally {
session.endSession()
}
}
I expect my code to add in the database and to update the entry packages in atomic way, that there is no instable database status.
This is working perfectly for the main part, but I need to be sure that this bug is not showing anymore.
You should use the session.withTransaction() helper function to perform the transaction, as pointed in mongoose documentation. This will take care of starting, committing and retrying the transaction in case it fails.
const session = await mongoose.startSession();
await session.withTransaction(async () => {
// Your transaction methods
});
Explanation:
The multi-document transactions in MongoDB are relatively new and might be a bit unstable in some cases, such as described here. And certainly, it has also been reported in Mongoose here. Your error most probably is a TransientTransactionError due to a write-conflict happening when the transaction is committed.
However, this is a known and expected issue from MongoDB and these comments explain their reasoning behind why they decided it to be like this. Moreover, they claim that the user should be handling the cases of write conflicts and retrying the transaction if that happens.
Therefore, looking at your code, the Package.create(...) method seems to be the reason why the error gets triggered, since this method is executing a save() for every document in the array (from mongoose docs).
A quick solution might be using Package.insertMany(...) instead of create(), since the Model.insertMany() "only sends one operation to the server, rather than one for each document" (from mongoose docs).
However, MongoDB provides a helper function session.withTransaction() that will take care of starting and committing the transaction and retry it in case of any error, since release v3.2.1. Hence, this should be your preferred way to work with transactions in a safer way; which is, of course, available in Mongoose through the Node.js API.
The accepted answer is great. In my case, I was running multiple transactions serially within a session. I was still facing this issue every now and then. I wrote a small helper to resolve this.
File 1:
// do some work here
await session.withTransaction(() => {});
// ensure the earlier transaction is completed
await ensureTransactionCompletion(session);
// do some more work here
await session.withTransaction(() => {});
Utils File:
async ensureTransactionCompletion(session: ClientSession, maxRetryCount: number = 50) {
// When we are trying to split our operations into multiple transactions
// Sometimes we are getting an error that the earlier transaction is still in progress
// To avoid that, we ensure the earlier transaction has finished
let count = 0;
while (session.inTransaction()) {
if (count >= maxRetryCount) {
break;
}
// Adding a delay so that the transaction get be committed
await new Promise(r => setTimeout(r, 100));
count++;
}
}
I am using knex npm version 0.15.2. while Rollback the transaction I'm getting the following error:
Error: Transaction rejected with non-error: undefined
Trx.rollback()
above function used for rollback.
Same code working for knex version 0.12.6
This is the function I used for commit/Rollback.
function Commit(pTrx, pIsCommit, pCallback) {
try {
var co = require("co");
var q = require('q');
var Q = q.defer();
co(function* () {
if (pIsCommit) {
yield pTrx.commit();
} else {
yield pTrx.rollback();
}
Q.resolve(pCallback('SUCCESS'));
}).catch(function (error) {
Q.reject(pCallback(error));
});
return Q.promise;
} catch (error) {
console.log(error)
}
}
This code could use some work. :) Here are a few things that pop out:
You don't need co or q anymore. Promises and async/await are built-in and much simpler to use. Async functions automatically return promises that will be resolved with the value returned or rejected if an error is thrown. Learn about async/await here: https://jsao.io/2017/07/how-to-get-use-and-close-a-db-connection-using-async-functions/
You shouldn't return success as a string. If the function completes without throwing an exception, then success is implied. I see people do this from time to time, but it's often for the wrong reasons.
You shouldn't accept callback in a function that returns a promise, it should be one or the other. The caller of your function will either pass a callback or await it's completion.
If you are going to use callbacks, then you should return null as the first parameter when successful. See the last sentence here: https://nodejs.org/en/knowledge/getting-started/control-flow/what-are-callbacks/
Your function name, Commit, starts with an uppercase. This convention is typically used to indicate that the function is a contstructor function and meant to be invoked with the new keyword.
Here's how the function could look once cleaned up:
async function commit(pTrx, pIsCommit) {
// Not using a try/catch. If an error is thrown the promise returned will be rejected.
if (pIsCommit) {
await pTrx.commit();
} else {
await pTrx.rollback();
}
// Not going to return anything. If we get to this point then success is implied when the promise is resolved.
}
A consumer of your function would call it with something like:
async function myWork() {
// do work; get pTrx
try {
await commit(pTrx, true);
// If I get here, then I can assume commit was successful, no need to check a return value of 'SUCCESS'
} catch (err) {
// handle error
}
}
It's hard to say where the issue is with the code in its current state. However, if the issue truly is with Knex, then you should probably post this as an issue in the Knex repo: https://github.com/tgriesser/knex/issues But you should write a reproducible test case that proves its an issue with Knex.
had following code which worked:
let sdk = new SDK({ name: "somevalue"});
module.exports= ()=> {
sdk.dothing();
}
I then needed to change the parameter to use data from an async function:
let asyncfunc = require('asyncfunc');
let sdk = new SDK({ name: (()=>{
return asyncfunc()
.then((data) => {
return data.value
})
});
module.exports= ()=> {
sdk.dothing();
}
Following the change, the call to new SDK is failing because the parameter passed is {} as the asyncFunc promise has not yet resolved.
I'm getting back into node after a year and new to promises. what is the proper way to do this?
As you've found, you can't pass in a promise to something that's expecting a string. You need to wait for the asynchronous operation to complete.
This means that your SDK won't be ready right away, so you have two options:
Change your module so it returns a promise for the needed value. Anyone who needs to use your module would need to use the returned promise.
Example:
let pSdk = asyncFunc()
.then(data => new SDK({ name: data.value }));
module.exports = () => pSdk.then(sdk => sdk.dothing());
Store an sdk value that's not populated immediately. Users of your module can obtain the SDK instance directly, but it might not be ready when they need it.
Example:
let sdk;
asyncFunc()
.then(data => sdk = new SDK({ name: data.value }));
module.exports = () => {
if(!sdk) { throw new Error("The SDK is not ready yet!"); }
return sdk.dothing();
};
if any bit of code, in node, is asynchronous then immediately next bit of code will be executed. it doesn't matter if the asynchronous code is wrapped in promise or not.( For codes wrapped in the promise the compiler will return a pending promise to be resoled or rejected and proceed to the next bit of code.) When you are creating an object using new SDK({ }) the name is having reference to a pending promise which is yet to be settled that's why your code is failing to fulfill your requirement. You can do it this way to resolve your problem.
asyncfunc()
.then((data) => {
return new SDK({ name: data.value });
}).then(function(sdk){
//do your work here using sdk
})
One important point to be noted here is you can't return from .then() to assign the value to any variable as you are doing. The value returned from .then() will be accessible from the next chained .then() not by outside global variable.Since you are exporting sdk.dothing() so you need to export it inside the last .then()