In our program, we try to implement a task retry pattern with await.
Our main problem is our method keeps the first retry payload in subsequent ones.
Here is the retry method:
async retryTaskUntilExpectedValue({
task,
expectedValue,
messageOnError = 'Max retry number reached without expected result',
maxRetries = 10,
timeout = 10,
spinner = null
}) {
let printFn = console.log;
if (spinner !== null) {
printFn = spinner.text;
}
// Proceed retries
for (let i = 1; i <= maxRetries; i++) {
try {
let result = await task;
console.log(result); // Always display same result: {"state": "upgrading"} even if curling returns {"state": "upgraded"} after about 2 retries
result = JSON.parse(result).state;
if (result === expectedValue) {
return Promise.resolve(result);
} else if (i <= maxRetries) {
printFn(`Result "${result}" differs from expected value "${expectedValue}"`);
await wait(1000);
printFn(`Waiting ${timeout}s before retry`);
await wait(timeout * 1000);
printFn(`Retrying (${i})`);
continue;
} else {
return Promise.reject(`ERROR: ${messageOnError}`);
}
} catch (err) {
return Promise.reject(`ERROR: Unexpected error while running task`);
}
}
};
And the use in our CLI:
checkUpgrade(url) {
return retryTaskUntilExpectedValue({
task: this.makeHttpRequest('GET', url),
expectedValue: 'upgraded'
});
}
In our case, the task is an http request returning a state from our backend database.
The model is simple:
{ "state": "upgrading" } then when the backend job is done, it returns { "state": "upgraded"}.
The job takes some time to process (around 20 sec). In our tests, this behavior occured:
First call: upgrading
First retry: upgrading
After that, by curling directly the REST api by hand, I get the upgraded status
all other retries in the CLI: upgrading
So in the CLI we build, we have 10 times the result: Result "upgrading" differs from expected value "upgraded"
It seems the let response = await task; in the subsequent retries does not call the task method at each retry. Indeed if actual call was made, it would for sure retrieve the proper upgraded state since we get it through curl.
How to make the await task; to actually trigger the call task method and not to keep the result from first call?
A promise is the result for an already started operation. By passing in task as a promise inside - it will always await the same result and return the same value.
Instead, retryTaskUntilExpectedValue should take a function for a promise and await an invocation of that:
let result = await functionReturningTask();
Where functionReturningTask is whatever you used to obtain task in the first place.
Related
I've written a transaction block in Postgresql (via node-postgres) and it's working fine, although I would like to ask if it's possible (and how) to put an if-else condition within the transaction block.
This is my current code (working as intended):
async function execute() {
// Promise chain for pg Pool client
const client = await pool
.connect()
.catch(err => {
console.log("\nclient.connect():", err.name);
process.exit();
});
//Initiate the Postgres transaction
await client.query("BEGIN");
try {
... <<constant declarations>>
// Pass SQL string to the query() method
await client.query(sqlString, sqlValues, function(err, result) {
<< Insert If-Else Condition 1 Here >>
<< Insert If-Else Condition 2 Here >>
if (someCondition == 1) {
// Rollback before executing another transaction
client.query("ROLLBACK");
} else if (err) {
client.query("ROLLBACK");
res.status(500).send("Server Error");
} else {
client.query("COMMIT");
res.json({ "message": "done!" });
}
});
} catch (er) {
// Rollback before executing another transaction
client.query("ROLLBACK");
}
} finally {
client.release();
}
execute();
The code above is working, although I want to put two if-else condition blocks in the transaction, which will fire 2 queries if the conditions were satisfied, and if whether true or false, will continue to the if (someCondition == 1) condition block.
These are the two if-else conditions that I want to put:
if (conditionA == true) {
await pool.query(query1)
}
if (conditionB == true) {
await pool.query(query2)
}
Running them causes an error:
SyntaxError: await is only valid in async function, with the origin being on await pool.query(query1).
Removing await causes an unresolved promise error.
I'm stumped on how to do this part. I've been tinkering with nested transactions via savepoint but to no avail.
Thank you in advanced!
"await is only valid in async function"
and your code is inside an async function?
async function execute() {
but... actually no, it is inside this one
await client.query(sqlString, sqlValues, function(err, result) {
^
here is the function---------------------^
...and this function is not async, which explains the error message.
I wonder why you're using a callback with client.query and going asynchronous at the same time, since the whole point of going asynchronous is to avoid those evil callbacks. Is this an oversight? Will it even work if you add "async"? Or maybe it's because I've never used node.js so I have no idea what I'm talking about.
Note:
if (someCondition == 1) {
// Rollback before executing another transaction
client.query("ROLLBACK");
I have no idea what this rollback is doing...
I know we can retry failed jobs in Bull using the backoff attribute in Bull like below
var Queue = require('bull');
var myQueue = new Queue('myQueue');
runQueryQueue
.add(data, {
jobId: uid,
attempts: 10,
backoff: {
type: 'exponential',
delay: 2000,
},
stackTraceLimit: 100,
})
.then((result) => {})
.catch(console.log);
But I want to retry only when the failure reasons are worth retrying - example timeouts, rate limits, etc and not retry when the errors are related to user inputs, etc.
How can I check the failure error message and decide to retry?
You can mark a job as complete, e.g. do not throw errors, but write error data to a result of job completion. Result can be processed different way, if there is any sign of error.
async function jobProcess(job) {
if (doNotRetryError) {
return doNotRetryError
} else if (anyOtherError) {
throw new Error('retry')
} else {
return {success: true}
}
}
async function jobCompleted(job, result) {
if (result instanceof Error) {
// some error happened, but job shouldn't be retried
} else {
// job completed
}
}
Keep setting your backoff config. If the job is not worth retrying, set opts.attempts to attemptsMade + 1 or smaller inside the processor.
Your changed opts will not be saved, but your job will be moved to the failed list instantly.
async process(job: Job): Promise<void> {
try {
await doStuff(job);
} catch (error) {
if (!isWorthRetrying(error)) {
// Move job to failed list instantly by setting attemps to attemptsMade + 1 or smaller
job.opts.attempts = job.attemptsMade + 1;
}
throw error;
}
}
This answer is for bull-mq so would be useful if you're willing to upgrade.
There is now an UnrecoverableError you can throw and bull-mq will take care of not retying the job.
https://docs.bullmq.io/guide/retrying-failing-jobs#stop-retrying-jobs
bull has a job.discard() function that should allow you to mark a job as non-retriable in the process.
I'm writing a Windows Node.js server app (using ES6 btw).
The first thing I want to do - in the top-level code - is sit in a while loop, calling an async function which searches for a particular registry key/value. This function is 'proven' - it returns the value data if found, or else throws:
async GetRegValue(): Promise<string> { ... }
I need to sit in a while loop until the registry item exists, and then grab the value data. (With a delay between retries).
I think I know how to wait for an async call to complete (one way or the other) before progressing with the rest of the start-up, but I can't figure out how to sit in a loop waiting for it to succeed.
Any advice please on how to achieve this?
(I'm fairly new to typescript, and still struggling to get my head round all async/await scenarios!)
Thanks
EDIT
Thanks guys. I know I was 'vague' about my code - I didn't want to put my real/psuedo code attempts, since they have all probably overlooked the points you can hopefully help me understand.
So I just kept it as a textual description... I'll try though:
async GetRegValue(): Promise<string> {
const val: RegistryItem = await this.GetKeyValue(this.KEY_SW, this.VAL_CONN);
return val.value
}
private async GetKeyValue(key: string, name: string): Promise<RegistryItem> {
return await new Promise((resolve, reject) => {
new this.Registry({
hive: this.Hive, key
}).get(name, (err, items) => {
if (err) {
reject(new Error('Registry get failed'));
}
else {
resolve( items );
}
});
})
.catch(err => { throw err });
}
So I want to do something like:
let keyObtained = false
let val
while (keyObtained == false)
{
// Call GetRegValue until val returned, in which case break from loop
// If exception then pause (e.g. ~100ms), then loop again
}
}
// Don't execute here till while loop has exited
// Then use 'val' for the subsequent statements
As I say, GetRegValue() works fine in other places I use it, but here I'm trying to pause further execution (and retry) until it does come back with a value
You can probably just use recursion. Here is an example on how you can keep calling the GetRegValue function until is resolves using the retryReg function below.
If the catch case is hit, it will just call GetRegValue over and over until it resolves successfully.
you should add a counter in the catch() where if you tried x amount of times you give up.
Keep in mind I mocked the whole GetRegValue function, but given what you stated this would still work for you.
let test = 0;
function GetRegValue() {
return new Promise((resolve, reject) => {
setTimeout(function() {
test++;
if (test === 4) {
return resolve({
reg: "reg value"
});
}
reject({
msg: "not ready"
});
}, 1000);
});
}
function retryReg() {
GetRegValue()
.then(registryObj => {
console.log(`got registry obj: ${JSON.stringify(registryObj)}`)
})
.catch(fail => {
console.log(`registry object is not ready: ${JSON.stringify(fail)}`);
retryReg();
});
}
retryReg();
I don't see why you need this line:
.catch(err => { throw err });
The loop condition of while isn't much use in this case, as you don't really need a state variable or expression to determine if the loop should continue:
let val;
while (true)
{
try {
val = await GetRegValue(/* args */);
break;
} catch (x) {
console.log(x); // or something better
}
await delay(100);
}
If the assignment to val succeeds, we make it to the break; statement and so we leave the loop successfully. Otherwise we jump to the catch block and log the error, wait 100 ms and try again.
It might be better to use a for loop and so set a sensible limit on how many times to retry.
Note that delay is available in an npm package of the same name. It's roughly the same as:
await new Promise(res => setTimeout(res, 100));
I am trying to call a generator function inside setInterval() method. The objective of this code is it will query a particular server for some data periodically, until it gets a non zero response. Upon getting the response it will call storeAddress() which is a generator function defined in the same file.
The below code is giving me an error like this:
SyntaxError: yield is a reserved word (248:6)
NOTE: I am using react-boilerplate to build my app. The above error is thrown by babel, as far as I can tell through searching internet.
I have tried const query = yeild call (setInterval, function(){magic code}, 10000). This does not give the error, but magic code never gets executed.
I have tried const query = setInterval(function* () {magic code}, 10000) with the same effect as above.
I have tried const query = setInterval(yield call(function(){magic code}, 10000) with same effect as above.
I have tried const query = yield call (setInterval, function(){magic code}, 10000) with same effect as above.
I have tried storeAddress(action.payload, balance).next() inside setInterval(). The control does flow inside storeAddress(), but that function also have generator calls inside, which never gets invoked. In fact nothing after the first generator call inside storeAddress() gets executed in this case.
function* callSaveNewAddress(action){
const selectedNetwork = yield select(selectNetworkId());
let count = 1;
let balance = 0;
const query = setInterval(function () {
getAddressBalance(action.payload, selectedNetwork).then(res =>
{return balance = res.data.mempool_balance});
if(balance > 0) {
yield call (storeAddress, action.payload, balance);
clearInterval(query);
} else if(count == 90) {
clearInterval(query);
console.log("Nothing received in 15 minutes");
}
}, 10000);
}
So how am I suppose to call storeAddress(), which is a generator function, inside a normal function like setInterval()?
const query= function * () {
const runner = yield call(setInterval, () => {
getAddressBalance(action.payload, selectedNetwork).then(res =>
{return balance = res.data.mempool_balance});
if(balance > 0) {
yield call (storeAddress, action.payload, balance);
clearInterval(query);
} else if(count == 90) {
clearInterval(query);
console.log("Nothing received in 15 minutes");
}
}, 1000);
}
try to use the setInterval within a call, passing through parameters the function you want to execute within it.
As per Understanding the node.js event loop, node.js supports a single thread model. That means if I make multiple requests to a node.js server, it won't spawn a new thread for each request but will execute each request one by one. It means if I do the following for the first request in my node.js code, and meanwhile a new request comes in on node, the second request has to wait until the first request completes, including 5 second sleep time. Right?
var sleep = require('sleep');
sleep.sleep(5)//sleep for 5 seconds
Is there a way that node.js can spawn a new thread for each request so that the second request does not have to wait for the first request to complete, or can I call sleep on specific thread only?
If you are referring to the npm module sleep, it notes in the readme that sleep will block execution. So you are right - it isn't what you want. Instead you want to use setTimeout which is non-blocking. Here is an example:
setTimeout(function() {
console.log('hello world!');
}, 5000);
For anyone looking to do this using es7 async/await, this example should help:
const snooze = ms => new Promise(resolve => setTimeout(resolve, ms));
const example = async () => {
console.log('About to snooze without halting the event loop...');
await snooze(1000);
console.log('done!');
};
example();
In case you have a loop with an async request in each one and you want a certain time between each request you can use this code:
var startTimeout = function(timeout, i){
setTimeout(function() {
myAsyncFunc(i).then(function(data){
console.log(data);
})
}, timeout);
}
var myFunc = function(){
timeout = 0;
i = 0;
while(i < 10){
// By calling a function, the i-value is going to be 1.. 10 and not always 10
startTimeout(timeout, i);
// Increase timeout by 1 sec after each call
timeout += 1000;
i++;
}
}
This examples waits 1 second after each request before sending the next one.
Please consider the deasync module, personally I don't like the Promise way to make all functions async, and keyword async/await anythere. And I think the official node.js should consider to expose the event loop API, this will solve the callback hell simply. Node.js is a framework not a language.
var node = require("deasync");
node.loop = node.runLoopOnce;
var done = 0;
// async call here
db.query("select * from ticket", (error, results, fields)=>{
done = 1;
});
while (!done)
node.loop();
// Now, here you go
When working with async functions or observables provided by 3rd party libraries, for example Cloud firestore, I've found functions the waitFor method shown below (TypeScript, but you get the idea...) to be helpful when you need to wait on some process to complete, but you don't want to have to embed callbacks within callbacks within callbacks nor risk an infinite loop.
This method is sort of similar to a while (!condition) sleep loop, but
yields asynchronously and performs a test on the completion condition at regular intervals till true or timeout.
export const sleep = (ms: number) => {
return new Promise(resolve => setTimeout(resolve, ms))
}
/**
* Wait until the condition tested in a function returns true, or until
* a timeout is exceeded.
* #param interval The frenequency with which the boolean function contained in condition is called.
* #param timeout The maximum time to allow for booleanFunction to return true
* #param booleanFunction: A completion function to evaluate after each interval. waitFor will return true as soon as the completion function returns true.
*/
export const waitFor = async function (interval: number, timeout: number,
booleanFunction: Function): Promise<boolean> {
let elapsed = 1;
if (booleanFunction()) return true;
while (elapsed < timeout) {
elapsed += interval;
await sleep(interval);
if (booleanFunction()) {
return true;
}
}
return false;
}
The say you have a long running process on your backend you want to complete before some other task is undertaken. For example if you have a function that totals a list of accounts, but you want to refresh the accounts from the backend before you calculate, you can do something like this:
async recalcAccountTotals() : number {
this.accountService.refresh(); //start the async process.
if (this.accounts.dirty) {
let updateResult = await waitFor(100,2000,()=> {return !(this.accounts.dirty)})
}
if(!updateResult) {
console.error("Account refresh timed out, recalc aborted");
return NaN;
}
return ... //calculate the account total.
}
It is quite an old question, and though the accepted answer is still entirely correct, the timers/promises API added in v15 provides a simpler way.
import { setTimeout } from 'timers/promises';
// non blocking wait for 5 secs
await setTimeout(5 * 1000);