I am working with Knex to write a simple transaction. I am inserting an object into one table and then insert an array of objects. Since I am using Knex in my API, I would like to know which insert failed. Unfortunately the error objects only shares this information:
{
"length":130,
"name":"error",
"severity":\"ERROR\",
"code":\"22P02\",
"file":\"uuid.c\",
"line":\"137\",
"routine":\"string_to_uuid\"
}
I would like to know though which query exactly failed to send back a proper error message to the Frontend. I am using a lambda setup with a RDS connection.
This is my transaction:
const test = await client.transaction(async (trx) => {
try {
await trx<any>('schema.table')
.insert({
id: answers.myId,
addition: 'some text',
});
const test = answers.answers.map((answer) => ({
id: answer.id,
my_id: answers.myId,
offered_answer_id: answer.offered_answer_id,
value: answer.value !== '' ? answer.value : null,
}));
await trx<any>('schema.table2').insert(test);
} catch (error) {
return error;
}
I am triggering an error by manipulating an uuid. So the error object is technically correct but I cannot figure out which insert caused it with this information.
Any help is very appreciated!
Related
I'm getting a little problem which I'm not being capable to debug. I wrote a little Firebase Function to get data from a JSON object and to store it in a Firestore Document. Simple.
It works, except the first time I run it after deployed (or after a long time has passed since the last execution). I have to run it once (without working), and then the subsequent tries always work, and I can see the new document being created with all the data inside it.
In the first attempt, there are no logs: Function execution took 601 ms, finished with status code: 200. Despite that, no document is being created nor changes being made.
In the second and subsequent attempts, If I request the function execution with a HTTP POST to https://cloudfunctions/functionName?id=12345, then the document '12345' is created inside collection with all the data inside it.
The collection where the documents are stored (scenarios) already exist in the database before any function call is executed.
This is the code:
const functions = require("firebase-functions");
const admin = require("firebase-admin");
admin.initializeApp();
const db = admin.firestore();
db.settings({ignoreUndefinedProperties: true});
const fetch = require("node-fetch");
let scenarioData;
const fetchScenarioJSON = async (scenarioId) => {
try {
const response = await fetch(`https://url/api/scenarios/single/${scenarioId}`);
const scenarioText = await response.text();
scenarioData = JSON.parse(scenarioText);
} catch (err) {
return ("not valid json");
}
return scenarioData;
};
/**
* Add data to Firestore.
* #param {JSON} scenario JSON array containing the scenario data.
*/
async function addDataToFirestore(scenario) {
const data = {
id: scenario.scenario._id,
name: scenario.scenario.name,
description: scenario.scenario.description,
language: scenario.scenario.language,
author: scenario.scenario.author,
draft: scenario.scenario.draft,
last_modified: scenario.scenario.last_modified,
__v: scenario.scenario.__v,
duration: scenario.scenario.duration,
grade: scenario.scenario.grade,
deleted: scenario.scenario.deleted,
view_count: scenario.scenario.view_count,
comments_count: scenario.scenario.comments_count,
favorites_count: scenario.scenario.favorites_count,
activities_duration: scenario.scenario.activities_duration,
activities: scenario.scenario.activities,
outcomes: scenario.scenario.outcomes,
tags: scenario.scenario.tags,
students: scenario.scenario.students,
created: scenario.scenario.created,
subjects: scenario.scenario.subjects,
};
const res = await db.collection("scenarios").doc(scenario.scenario._id).set(data);
}
exports.functionName =
functions.https.onRequest((request, response) => {
return fetchScenarioJSON(request.query.id).then((scenario) => {
if (typeof scenario === "string") {
if (scenario.includes("not valid json")) {
response.send("not valid json");
}
} else {
addDataToFirestore(scenario);
response.send(`Done! Added scenario with ID ${request.query.id} to the app database.`);
}
});
});
My question is if I am doing anything wrong with the code that makes the execution not work on the first call after it is deployed, but actually does work in subsequent calls.
It is most probably because you don't wait that the asynchronous addDataToFirestore() function is completed before sending back the response.
By doing
addDataToFirestore(scenario);
response.send()
you actually indicate (with response.send()) to the Cloud Function platform that it can terminate and clean up the Cloud Function (see the doc for more details). Since you don't wait for the asynchronous addDataToFirestore() function to complete, the doc is not written to Firestore.
The "erratic" behaviour (sometimes it works, sometimes not) can be explained as follows:
In some cases, your Cloud Function is terminated before the write to Firestore is fully executed, as explained above.
But, in some other cases, it may be possible that the Cloud Functions platform does not immediately terminate your CF, giving enough time for the write to Firestore to be fully executed. This is most probably what happens after the first call: the instance of the Cloud Function is still running and then the docs are written with the "subsequent calls".
The following modifications should do the trick (untested). I've refactored the Cloud Function with async/await, since you use it in the other functions.
// ....
async function addDataToFirestore(scenario) {
const data = {
id: scenario.scenario._id,
name: scenario.scenario.name,
description: scenario.scenario.description,
language: scenario.scenario.language,
author: scenario.scenario.author,
draft: scenario.scenario.draft,
last_modified: scenario.scenario.last_modified,
__v: scenario.scenario.__v,
duration: scenario.scenario.duration,
grade: scenario.scenario.grade,
deleted: scenario.scenario.deleted,
view_count: scenario.scenario.view_count,
comments_count: scenario.scenario.comments_count,
favorites_count: scenario.scenario.favorites_count,
activities_duration: scenario.scenario.activities_duration,
activities: scenario.scenario.activities,
outcomes: scenario.scenario.outcomes,
tags: scenario.scenario.tags,
students: scenario.scenario.students,
created: scenario.scenario.created,
subjects: scenario.scenario.subjects,
};
await db.collection("scenarios").doc(scenario.scenario._id).set(data);
}
exports.functionName =
functions.https.onRequest(async (request, response) => {
try {
const scenario = await fetchScenarioJSON(request.query.id);
if (typeof scenario === "string") {
if (scenario.includes("not valid json")) {
response.send("not valid json");
}
} else {
await addDataToFirestore(scenario); // See the await here
response.send(`Done! Added scenario with ID ${request.query.id} to the app database.`);
}
} catch (error) {
// ...
}
});
I am using node-pg with typescript.
I have a getPool utility from the doc https://node-postgres.com/features/pooling
export const getPool = (config?: PoolConfig) => {
const pool = new Pool(config);
pool.on('error', (err, client) => {
console.error('Unexpected error on idle client', err);
process.exit(-1);
});
return pool;
};
I use it like this in an async/await context
const pool = getPool();
await pool.query('my sql query here...');
When I have an invalid SQL query I get this kind of error:
error: null value in column "foo" violates not-null constraint
at Parser.parseErrorMessage (node_modules/pg-protocol/src/parser.ts:357:11)
at Parser.handlePacket (node_modules/pg-protocol/src/parser.ts:186:21)
at Parser.parse (node_modules/pg-protocol/src/parser.ts:101:30)
at Socket.<anonymous> (node_modules/pg-protocol/src/index.ts:7:48)
Note: I would understand if it was the pool.on('error')'s callback that stole my stack trace, but the errors are not prefixed with Unexpected error on idle client
Notice in the stack trace there is no line that points to a file in my codebase.
My problem is, I have hundereds of queries in my codebase, and I would like to be able to trace the line that called the failing pool.query(). This would help a lot to find which query triggered the error.
Expected :
error: null value in column "foo" violates not-null constraint
at ...
at mycodebase/src/myfile.ts:42
I use dirty hack (patch Pool.prototype), but it works for me:
const originalPoolQuery = Pool.prototype.query;
Pool.prototype.query = async function query(...args) {
try {
return await originalPoolQuery.apply(this, args);
} catch (e) {
// All magic is here. new Error will generate new stack, but message will copyid from e
throw new Error(e)
}
}
// After hack create pool and good luck
const pool = new Pool({})
await pool.query('SELECT * FROM ...')
Stacktrace in this case will be more informative.
I think that pool.on('error', cb) is not for catching query errors, it for connection errors (i am not sure)
When working with a big application that has several tables and several DB operations it's very difficult to keep track of what transactions are occurring. To workaround this we started by passing around a trx object.
This has proven to be very messy.
For example:
async getOrderById(id: string, trx?: Knex.Transaction) { ... }
Depending on the function calling getOrderById it will either pass a trx object or not. The above function will use trx if it is not null.
This seems simple at first, but it leads to mistakes where if you're in the middle of a transaction in one function and call another function that does NOT use a transaction, knex will hang with famous Knex: Timeout acquiring a connection. The pool is probably full.
async getAllPurchasesForUser(userId: string) {
..
const trx = await knex.transaction();
try {
..
getPurchaseForUserId(userId); // Forgot to make this consume trx, hence Knex timesout acquiring connection.
..
}
Based on that, I'm assuming this is not a best practice, but I would love if someone from Knex developer team could comment.
To improve this we're considering to instead use knex.transactionProvider() that is accessed throughout the app wherever we perform DB operations.
The example on the website seems incomplete:
// Does not start a transaction yet
const trxProvider = knex.transactionProvider();
const books = [
{title: 'Canterbury Tales'},
{title: 'Moby Dick'},
{title: 'Hamlet'}
];
// Starts a transaction
const trx = await trxProvider();
const ids = await trx('catalogues')
.insert({name: 'Old Books'}, 'id')
books.forEach((book) => book.catalogue_id = ids[0]);
await trx('books').insert(books);
// Reuses same transaction
const sameTrx = await trxProvider();
const ids2 = await sameTrx('catalogues')
.insert({name: 'New Books'}, 'id')
books.forEach((book) => book.catalogue_id = ids2[0]);
await sameTrx('books').insert(books);
In practice here's how I'm thinking about using this:
SingletonDBClass.ts:
const trxProvider = knex.transactionProvider();
export default trxProvider;
Orders.ts
import trx from '../SingletonDBClass';
..
async getOrderById(id: string) {
const trxInst = await trx;
try {
const order = await trxInst<Order>('orders').where({id});
trxInst.commit();
return order;
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to fetch order, error: ${e}`);
}
}
..
Am I understanding this correctly?
Another example function where a transaction is actually needed:
async cancelOrder(id: string) {
const trxInst = await trx;
try {
trxInst('orders').update({ status: 'CANCELED' }).where({ id });
trxInst('active_orders').delete().where({ orderId: id });
trxInst.commit();
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to cancel order, error: ${e}`);
}
}
Can someone confirm if I'm understanding this correctly? And more importantly if this is a good way to do this. Or is there a best practice I'm missing?
Appreciate your help knex team!
No. You cannot have global singleton class returning the transaction for your all of your internal functions. Otherwise you are trying always to use the same transaction for all the concurrent users trying to do different things in the application.
Also when you once commit / rollback the transaction returned by provider, it will not work anymore for other queries. Transaction provider can give you only single transaction.
Transaction provider is useful in a case, where you have for example middleware, which provides transaction for request handlers, but it should not be started, since it might not be needed so you don't want yet allocate a connection for it from pool.
Good way to do your stuff is to pass transcation or some request context or user session around, so that each concurrent user can have their own separate transactions.
for example:
async cancelOrder(trxInst, id: string) {
try {
trxInst('orders').update({ status: 'CANCELED' }).where({ id });
trxInst('active_orders').delete().where({ orderId: id });
trxInst.commit();
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to cancel order, error: ${e}`);
}
}
Depending on the function calling getOrderById it will either pass a trx object or not. The above function will use trx if it is not null.
This seems simple at first, but it leads to mistakes where if you're in the middle of a transaction in one function and call another function that does NOT use a transaction, knex will hang with famous Knex: Timeout acquiring a connection. The pool is probably full.
We usually do it in a way that if trx is null, query throws an error, so that you need to explicitly pass either knex / trx to be able to execute the method and in some methods trx is actually required to be passed.
Anyhow if you really want to force everything to go through single transaction in a session by default you could create API modules in a way that for each user session you create an API instance which is initialized with transaction:
const dbForSession = new DbService(trxProvider);
const users = await dbForSession.allUsers();
and .allUsers() does something like return this.trx('users');
I am using Mongoose to access to my database. I need to use transactions to make an atomic insert-update.
95% of the time my transaction works fine, but 5% of the time an error is showing :
"Given transaction number 1 does not match any in-progress transactions"
It's very difficult to reproduce this error, so I really want to understand where it is coming from to get rid of it.
I could not find a very clear explanation about this type of behaviour.
I have tried to use async/await key words on various functions. I don't know if an operation is not done in time or too soon.
Here the code I am using:
export const createMany = async function (req, res, next) {
if (!isIterable(req.body)) {
res.status(400).send('Wrong format of body')
return
}
if (req.body.length === 0) {
res.status(400).send('The body is well formed (an array) but empty')
return
}
const session = await mongoose.startSession()
session.startTransaction()
try {
const packageBundle = await Package.create(req.body, { session })
const options = []
for (const key in packageBundle) {
if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
options.push({
updateOne: {
filter: { _id: packageBundle[key].id },
update: {
$set: {
custom_id_string: 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
minimumIntegerDigits: 14,
useGrouping: false
})
},
upsert: true
}
}
})
}
}
await Package.bulkWrite(
options,
{ session }
)
for (const key in packageBundle) {
if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
packageBundle[key].custom_id_string = 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
minimumIntegerDigits: 14,
useGrouping: false
})
}
}
res.status(201).json(packageBundle)
await session.commitTransaction()
} catch (error) {
res.status(500).end()
await session.abortTransaction()
throw error
} finally {
session.endSession()
}
}
I expect my code to add in the database and to update the entry packages in atomic way, that there is no instable database status.
This is working perfectly for the main part, but I need to be sure that this bug is not showing anymore.
You should use the session.withTransaction() helper function to perform the transaction, as pointed in mongoose documentation. This will take care of starting, committing and retrying the transaction in case it fails.
const session = await mongoose.startSession();
await session.withTransaction(async () => {
// Your transaction methods
});
Explanation:
The multi-document transactions in MongoDB are relatively new and might be a bit unstable in some cases, such as described here. And certainly, it has also been reported in Mongoose here. Your error most probably is a TransientTransactionError due to a write-conflict happening when the transaction is committed.
However, this is a known and expected issue from MongoDB and these comments explain their reasoning behind why they decided it to be like this. Moreover, they claim that the user should be handling the cases of write conflicts and retrying the transaction if that happens.
Therefore, looking at your code, the Package.create(...) method seems to be the reason why the error gets triggered, since this method is executing a save() for every document in the array (from mongoose docs).
A quick solution might be using Package.insertMany(...) instead of create(), since the Model.insertMany() "only sends one operation to the server, rather than one for each document" (from mongoose docs).
However, MongoDB provides a helper function session.withTransaction() that will take care of starting and committing the transaction and retry it in case of any error, since release v3.2.1. Hence, this should be your preferred way to work with transactions in a safer way; which is, of course, available in Mongoose through the Node.js API.
The accepted answer is great. In my case, I was running multiple transactions serially within a session. I was still facing this issue every now and then. I wrote a small helper to resolve this.
File 1:
// do some work here
await session.withTransaction(() => {});
// ensure the earlier transaction is completed
await ensureTransactionCompletion(session);
// do some more work here
await session.withTransaction(() => {});
Utils File:
async ensureTransactionCompletion(session: ClientSession, maxRetryCount: number = 50) {
// When we are trying to split our operations into multiple transactions
// Sometimes we are getting an error that the earlier transaction is still in progress
// To avoid that, we ensure the earlier transaction has finished
let count = 0;
while (session.inTransaction()) {
if (count >= maxRetryCount) {
break;
}
// Adding a delay so that the transaction get be committed
await new Promise(r => setTimeout(r, 100));
count++;
}
}
I got the following error when trying to batch delete a collection:
TypeError: Cannot read property 'seconds' of null
at Function.fromProto (..\node_modules\#google-cloud\firestore\build\src\timestamp.js:91:47)
The code is plain simple (written in TypeScript):
var admin = require('firebase-admin');
...
var batch: FirebaseFirestore.WriteBatch = admin.firestore().batch();
return generic.application.admin.firestore().collection(name)
.get()
.then((docs: FirebaseFirestore.QuerySnapshot) => {
docs.docs.forEach((doc:any) => {
batch.delete(doc.ref);
});
})
.then(() => {
return batch.commit();
});
Happens when the collection is emtpy;
I solved by adding a counter so i can avoid commit when the collection is empty.
The issue was found firebase admin sdk version 5.13.0 but still present on version 5.13.1. More information can be found here.