I have defined a promise like this ...
const result = await Promise.race([
new Promise(resolve => {
consumer.run({
eachMessage: ({ message }) => {
const data = JSON.parse(message.value.toString());
if (data.payload.template
&& data.payload.template.id === '...'
&& data.payload.to[0].email === email) {
console.log('Should resolve!')
resolve(data.payload.template.variables.link);
console.log('resolved');
consumer.pause();
consumer.disconnect();
}
},
});
}),
new Promise((_, reject) => setTimeout(reject, 3000))
]);
console.log('result is ', result);
return result;
I can get to resolved but it doesnt print the result at the end, it seems like neither did the timeout nor the actual promise work as expected? Why is that? I suspect its something to do with using resolve inside the kafka js callback?
UPDATE: Seems like its Promise.race() thats not resolving, but why?
My suspicion is that your "success-side" promise inadvertently throws and you're swallowing the error silently.
Using a mock-up minimal implementation of the consumer (that succeeds or fails 50/50), the following code works.
Run the code sample a couple of times to see both cases.
var consumer = {
interval: null,
counter: 0,
run: function (config) {
this.interval = setInterval(() => {
this.counter++;
console.log(`Consumer: message #${this.counter}`);
config.eachMessage({message: this.counter});
}, 250);
},
pause: function () {
console.log('Consumer: paused');
clearInterval(this.interval);
},
disconnect: function () {
console.log('Consumer: disconnected');
}
};
Promise.race([
new Promise(resolve => {
const expectedMsg = Math.random() < 0.5 ? 3 : 4;
consumer.run({
eachMessage: ({ message }) => {
if (message === expectedMsg) resolve("success");
}
});
}),
new Promise((_, reject) => setTimeout(() => {
reject('timeout');
consumer.pause();
consumer.disconnect();
}, 1000))
]).then((result) => {
console.log(`Result: ${result}`);
}).catch((err) => {
console.log(`ERROR: ${err}`);
});
I have also moved consumer.pause() and consumer.disconnect() to the "timeout-side" promise, this way the consumer is guaranteed to disconnect, albeit it might run a tiny bit longer than necessary in the success case.
Related
I have a firebase v2 function set to timeout in 540 seconds however httpsCallableFromURL catches with an error saying Firebase Error: deadline-exceeded before the function finishes.
Here's the template of my firebase function:
exports.functionname = functions.https.onCall(
{
timeoutSeconds: 540,
cpu: 2,
},
async (req) => {
const getOutput = async (id: string) => {
return new Promise((resolve, reject) => {
return axios.get('ur'l)
.then((res) => {
resolve(res.data)
})
.catch((e) => {
resolve(false)
})
})
}
let output: any
const recurse = async () => {
return new Promise(async (resolve) => {
// first try getting output
output = await getOutput(id).then((res) => {
return res
})
// check for output
if (output.output) {
return resolve(true)
} else if (output.error) {
return resolve(false)
} else {
setTimeout(() => {
return resolve(recurse())
}, 2000)
}
})
}
return recurse()
.then(async () => {
const imageUrl = output.output[0]
function getBase64(url: string) {
return axios
.get(url, {
responseType: 'arraybuffer',
})
.then((response) =>
Buffer.from(response.data, 'binary').toString('base64'),
)
}
const base64Image = await getBase64(imageUrl)
const data = {
type: 'image',
predictionId,
result: base64Image,
}
return data
}
})
.catch((e) => {
return false
})
},
)
If I monitor the function logs it ends up finishing within the 540 seconds but my client never receives the return value since the httpsCallableFromURL errored out.
This issue involves some debugging with calculating execution time for each Promise because your Error is related to the timeout that is specified in the function which is 540 sec means all of your promises should be resolved or rejected in the given 540 secs.
So I will recommend you to place some console.time(“Promise 1”) and console.timeEnd(“Promise 1”) around each Promises so we can get expected execution time for each promise and compare the total execution time of function and with this data you can modify the the corresponding promise which takes a lot of time. For more about this topic go through this thread
And In the end If above all doesn’t work then you can just increase the timeout to 1800 sec means 30 mins to complete all the execution within the given timeout, but this should be avoided as a best practice.
Changing the memory limit to 16GiB and cpu to 4 seems to have solved this issue:
exports.functionname = functions.https.onCall(
{
timeoutSeconds: 540,
cpu: 4,
memory: '16GiB',
},
async (req) => {
},
)
I am trying to run a function asynchronously 20 times.
I have the function definition:
import axios from 'axios';
async function updateUser (url, user) {
return new Promise((resolve, reject) => {
axios.put(url, JSON.stringify(user))
.then(response => {
//Folio returned code 204
console.log('The user has been updated');
resolve(response.data);
}).catch((err) => {
//Folio returned error
console.error(`Error Text: ${err.response.data}`);
console.error(`Error Code: ${err}`);
});
});
};
export {updateUser};
Then I am trying to loop through users in an external JSON, and update them to my system 20 at a time:
import {updateUser} from './updateUser.js';
import usersjson from './jsons/users.json';
let promises=[];
if (usersjson.users.length) {
try {
for (const user of usersjson.users) {
update = new promise ((resolve,reject) => {
updateUser(url, user)
.then((list) => {resolve(list)})
.catch((error)=>{reject(error)})
});
promises.push(update);
if (promises.length = 20) {
await Promise.all(promises)
.then((responses)=> {
console.log(responses);
promises=[];
}).catch((err)=> {
console.log(err);
});
}
}
} catch(err)=> {
console.log(err);
}
}
However, I am getting an error message in the console:
await Promise.all(promises)
^^^^^
SyntaxError: Unexpected reserved word
My node version is 12.19.0 (based on previous questions, it seems it has to be at least 10, so that shouldn't be the issue).
When I run the function without await, they work correctly (i.e. the users are updated on my system).
It also seems that the for loop is asynchronous by itself, but I'm afraid that if there will be too many calls simultaneously my system will block the API, so I want to make 20-50 calls at a time, tops.
In order to use await you need to declare the function in which you are putting the await as async
For example:
const functionReturningPromise = (input) => {
return new Promise((resolve, reject) => {
if (!input) {
return reject();
}
return resolve();
});
}
const functionAwaitingPromise = async () => {
for (let i = 0; i < 20; i +=1) {
try {
await functionReturningPromise(i);
console.log(i);
} catch (error) {
console.log(error);
}
}
}
Problem: A library function is called, which sometimes does not work. If this function does not work, it sometimes happens that it does not return an error message but is stuck in an infinite loop.
Idea: My idea now would have been to use a timer, which executes a throw instruction after a given time, if the function has not been completed successfully before.
If the function was successfully completed, before the end of the timer, the timer should simply be closed.
Not working example code:
var errorCounter = 0;
async function a(m) {
try {
var myVar = setTimeout(function () {
throw console.log('Error');
}, 60000);
await libraryFunction(m);
clearTimeout(myVar);
} catch {
errorCounter++;
console.log(errorCounter);
}
}
You could use Promise.race to simulate a timeout (play with the numerical values below):
const timeout = new Promise((_, reject) => {
setTimeout(() => reject(new Error('timeout')), 1000);
});
const task = new Promise((resolve, _) => {
setTimeout(() => resolve('done'), 2000);
});
Promise.race([timeout, task])
.then((result) => { console.log(result); })
.catch((err) => { console.log(err.message); });
So in your case, it would look something like this:
const timeout = new Promise((_, reject) => {
setTimeout(() => reject(new Error('timeout')), 60 * 1000);
});
var errorCounter = 0;
function a(m) {
Promise.race([timeout, libraryFunction(m)])
.then((res) => { console.log(res); })
.catch((err) => {
errorCounter++;
console.log(errorCounter);
});
}
You can use Promise.race for this type of thing:
const MAX_WAIT_TIME = 500
const libFn = new Promise((resolve, reject) => {
setTimeout(() => resolve('success'), Math.random() * 1000);
})
const throwFn = new Promise((resolve, reject) => {
setTimeout(() => reject('failure'), MAX_WAIT_TIME);
})
;(async () => {
const result = await Promise.race([libFn, throwFn])
console.log('done') // this line won't execute if the timeout is reached
})()
I'm trying to develop a relative simple test on NodeJS 11.6.x. I'm not really a developer, but sometimes try to do some coding.
My objective is to create a SQLite database and repeat some steps every time a run the code:
1. Drop a table if it exists
2. Create a table
3. Insert N lines
4. Count how many lines is in the database
5. Close the database
I tried first with a basic approach using callback, but couldn't figure out a way to make the step 3 (insert N lines) and looking for a solution, the promise async/await 'pattern' sounded the way to accomplish everything.
But, after refactoring the code, the step 1 (drop table) isn't running and I still not being able to execute step 3 (insert N lines) and have no idea what is happening. I also tried to use a promise package with no luck.
Could someone please have a look and help on this and if possible, explain and or give some advice?
Thanks in advance
Edited: Well, I'not used to post here at SO and don't know the 'right' way to update things here. I beleave I should had left the first code as reference, buy I don't have anymore.
Now I think I'm almoust there. All steps are executing in order. Just the step 3 (insert N lines) that I'm not able to make it work. Or it inserts and stops not going to the next '.then' or it just insert 1 line and I'm cannot visualize what is happening.
In the code, I commented in two lines with 'BUG 1:' and 'BUG 2:'.
If I both are commented, I get what is happening, it inserts only 1 line and don't continue the promise chain
If I comment BUG 1 and let BUG 2 active, it inserts just one line and continues. I think I understand why
If I comment BUG 2 and let BUG 1 active, it inserts all lines but don't continue and again, I think I understand why
If I uncomment both (the way I think should work. Don't work, and return an aditional error "Segmentation fault"
Bellow the code:
const sqlite3 = require('sqlite3')
let db = new sqlite3.Database('./test.db');
waitTime = 1
process.stdout.write('Starting...\n')
var test = new Promise((resolve, reject) => {
process.stdout.write('Drop Table... ');
db.run(`DROP TABLE IF EXISTS test`, (err) => {
if (err) {
process.stdout.write(`Dropping Error ${err.message}\n`)
reject()
} else {
setTimeout(() => {
process.stdout.write(`Dropped!\n`)
resolve()
}, waitTime)
}
})
})
test.then(() => {
return new Promise((resolve, reject) => {
process.stdout.write('Create Table... ')
db.run(`CREATE TABLE IF NOT EXISTS test (data TEXT)`, (err) => {
if (err) {
process.stdout.write(`Creating Error ${err.message}\n`)
reject()
} else {
setTimeout(() => {
process.stdout.write(`Created!\n`)
resolve()
}, waitTime)
}
})
})
}).then(() => {
return new Promise((resolve, reject) => {
process.stdout.write('Insert Line... ')
lines = 10
let loop = (async () => {
for (let i = 0; i < lines; i++) {
await new Promise(resolve =>
db.run(`INSERT INTO test (data) VALUES ('a')`, (err) => {
if (err) {
process.stdout.write(`Inserting Error ${err.message}\n`)
throw (err)
} else {
setTimeout(() => {
// process.stdout.write(`Line ${i} Inserted!\n`)
process.stdout.write(`, ${i+1}`)
resolve() // BUG 1: if this line is commented, comment it, it will insert only 1 line
}, waitTime)
}
})
)
}
})()
process.stdout.write(`, IDone\n`)
resolve() // BUG 2: If this line is commented, the promise chain stops here
})
}).then(() => {
return new Promise((resolve, reject) => {
process.stdout.write('Count Line(s)... ')
db.all(`SELECT COUNT(*) AS totalLines FROM test`, [], (err, rows) => {
if (err) {
process.stdout.write(`Count Error ${err.message}\n`)
reject()
} else {
setTimeout(() => {
process.stdout.write(` ${rows[0].totalLines} Count!\n`)
resolve()
}, waitTime)
}
})
})
}).then(() => {
return new Promise((resolve, reject) => {
process.stdout.write('Select Line(s)... ')
db.all('SELECT data FROM test', [], (err, rows) => {
if (err) {
process.stdout.write(`Select Error ${err.message}\n`)
reject()
} else {
rows.forEach((row) => {
console.log(row.data);
})
setTimeout(() => {
process.stdout.write(`${rows[0].totalLines} Select!\n`)
resolve()
}, waitTime)
}
})
})
}).then(() => {
return new Promise((resolve, reject) => {
process.stdout.write('Close DB... ')
db.close((err) => {
if (err) {
process.stdout.write(`Closing Error ${err.message}\n`)
reject()
} else {
setTimeout(() => {
process.stdout.write(`Closed!\n`)
resolve()
}, waitTime)
}
})
})
}).then(() => {
console.log('Finished')
})
After the great explanation from #CertainPerformance (Thanks a lot), I was able to get it running. I believe it is now "the right" way to do it. May be there are some better ways, but for now, it is ok for me, bellow the final code:
const sqlite3 = require('sqlite3')
let db = new sqlite3.Database('./test.db');
lines = 10
process.stdout.write('Starting... ')
var test = new Promise((resolve, reject) => { process.stdout.write(`Promise Created...!\n`)
resolve()
})
test.then(() => { process.stdout.write('Drop Table... ')
return new Promise((resolve, reject) => {
db.run(`DROP TABLE IF EXISTS test`, (err) => {
if (err) {
reject(err)
} else { process.stdout.write(`Dropped!\n`)
resolve() }
})
})
}).then(() => { process.stdout.write('Create Table... ')
return new Promise((resolve, reject) => {
db.run(`CREATE TABLE IF NOT EXISTS test (data TEXT)`, (err) => {
if (err) {
reject(err)
} else {
process.stdout.write(`Created!\n`)
resolve() }
})
})
}).then(() => { process.stdout.write('Insert Line... ')
let insertLoop = (async () => {
for (let i = 0; i < lines; i++) {
await new Promise(resolve =>
db.run(`INSERT INTO test (data) VALUES ('a')`, (err) => {
if (err) {
reject(err)
} else { ( i == 0 ) ? process.stdout.write(`${i + 1}`) : process.stdout.write(`, ${i + 1}`)
resolve() }
})
)
}
process.stdout.write(`, Inserted!\n`)
})()
return insertLoop
}).then(() => { process.stdout.write('Count Line(s)... ')
return new Promise((resolve, reject) => {
db.all(`SELECT COUNT(*) AS totalLines FROM test`, [], (err, rows) => {
if (err) {
reject(err)
} else { process.stdout.write(` ${rows[0].totalLines} Counted!\n`)
resolve()
}
})
})
}).then(() => { process.stdout.write('Close DB... ')
return new Promise((resolve, reject) => {
db.close((err) => {
if (err) {
reject(err)
} else { process.stdout.write(`Closed!\n`)
resolve()
}
})
})
}).then(() => {
console.log('Finished')
}).catch((err) => {
process.stdout.write(`The process did not finish successfully: ${err}`)
})
There are two main issues. First, in the second .then, you declare loop as an async function that is immediately invoked: this means that loop will resolve to a Promise. The trimmed code looks like:
}).then(() => {
return new Promise((resolve, reject) => {
let loop = (async () => {
// do some asynchronus stuff
})()
resolve() // BUG 2
})
}).then(() => {
Declaring a Promise alone will not cause the current thread to wait for it. The above code doesn't work as expected for the same reason that this code prints after immediately:
console.log('start');
const prom = new Promise((resolve) => {
setTimeout(resolve, 500);
});
console.log('after');
You must call .then on a Promise (or await the Promise) in order to schedule additional operations after the Promise completes. Or, if you're currently inside a .then, you can return the Promise, which will mean that the next .then will run as soon as the returned Promise resolves:
}).then(() => {
let loop = (async () => {
// do some asynchronus stuff
})();
return loop;
}).then(() => {
// this block will run once `loop` resolves
Note the lack of a new Promise((resolve... constructor above - inside a .then, just returning the next Promise is often the preferred way to go, since it means a lot less code and avoids an antipattern.
The other issue with the current code is that errors will not be caught. For example, if your
db.run(`INSERT INTO test (data) VALUES ('a')`, (err) => {
if (err) {
process.stdout.write(`Inserting Error ${err.message}\n`)
throw (err)
// else call resolve()
throws an error, the Promise that's currently being awaited at that point will never resolve, nor will it reject - it will remain pending and unfulfilled forever. You should pass reject as the second argument to the Promise constructor, and call it when there's an error (instead of throw), for example:
await new Promise((resolve, reject) => {
db.run(`INSERT INTO test (data) VALUES ('a')`, (err) => {
if (err) {
process.stdout.write(`Inserting Error ${err.message}\n`)
reject(err)
} else {
// ...
This way, the awaited Promise will get rejected, which means that the whole loop will reject, and if the loop is returned, it'll allow a .catch to catch the error, for example:
var test = new Promise((resolve, reject) => {
// ...
});
test.then(() => {
return new Promise(...
// ...
})
.then(() => {
return new Promise(...
// ..
})
.then(() => {
return new Promise(...
// ..
})
.catch((err) => {
process.stdout.write(`The process did not finish successfully:`, err)
// handle errors
});
Note that, unless each db. function call needs to execute serially, it would be better to make all the requests at once, and resolve once each request has finished - this can significantly reduce the time required for the script to run. Create an array of Promises for each asynchronous call, then call Promise.all on that array to get a Promise that resolves when all of those Promises are done (or, rejects as soon as one of those Promises rejects). For example, for the second .then:
}).then(() => {
process.stdout.write('Insert Line... ')
const proms = Array.from(
{ length: lines },
(_, i) => new Promise((resolve, reject) => {
db.run(`INSERT INTO test (data) VALUES ('a')`, (err) => {
if (err) {
process.stdout.write(`Inserting Error ${err.message}\n`)
reject(err)
} else {
setTimeout(() => {
// process.stdout.write(`Line ${i} Inserted!\n`)
process.stdout.write(`, ${i+1}`)
resolve()
}, waitTime);
}
});
})
);
return Promise.all(proms);
}).then(() => {
Nothing else in your code looks to deal with asynchronous loops, luckily.
You may also consider a utility function like Promisify which will turn callback-based functions to Promises without all the extra new Promise(... boilerplate every time there's an asynchronous call.
Further improvements can be made by promisifying db into reusable functions, and leveraging the full power of async/await instead of mixing it with then:
const sqlite3 = require('sqlite3')
let db = new sqlite3.Database('./test.db');
function runDbAsync(sql) {
return new Promise((resolve, reject) => {
db.run(sql, (err) => {
if (err) reject(err);
else resolve();
});
});
}
function getDbAsync(sql, val) {
return new Promise((resolve, reject) => {
db.all(`SELECT COUNT(*) AS totalLines FROM test`, [], (err, rows) => {
if (err) reject(err);
else resolve(rows);
});
});
}
function closeDbAsync() {
return new Promise((resolve, reject) => {
db.close((err) => {
if (err) reject(err);
else resolve();
});
});
}
function write(text) {
process.stdout.write(text);
}
function writeLn(text) {
write(text + "\n");
}
async function main() {
const lines = 10
writeLn('Starting... ')
write('Drop Table... ');
await runDbAsync(`DROP TABLE IF EXISTS test`);
writeLn(`Dropped!`);
write('Create Table... ');
await runDbAsync(`CREATE TABLE IF NOT EXISTS test (data TEXT)`);
writeLn(`Created!`);
write('Insert Line... ');
for (let i = 0; i < lines; i++) {
await runDbAsync(`INSERT INTO test (data) VALUES ('a')`);
write( i == 0 `${i + 1}` : `, ${i + 1}`);
}
writeLn(`, Inserted!`);
write('Count Line(s)... ')
const rows = getDbAsync(`SELECT COUNT(*) AS totalLines FROM test`, []);
writeLn(` ${rows[0].totalLines} Counted!`)
write('Close DB... ');
await closeDbAsync();
writeLn(`Closed!`);
}
main().then(() => {
console.log('Finished')
}, err => {
writeLn(`The process did not finish successfully: ${err}`)
});
I have a problem testing ldapjs client search operation. It returns an EventEmitter that you have to make listen for some specific event. I wrapped this operations to promisify it and to define my logic and I would like to unit-test it.
findUser(username) {
return new Promise((resolve, reject) => {
logger.debug('Searching user: ', username);
this.ldapClient.bind(user.name, .user.password, err => {
if (err) return reject(err);
else
this.ldapClient.search(root, {filter: `(cn=${username})`}, (errSearch, resSearch) => {
if (errSearch) return reject(errSearch);
const entries = [];
resSearch.on('searchEntry', entry => entries.push(entry.object));
resSearch.on('searchReference', referral => reject(new Error(`Received search referall: ${referral}`)));
resSearch.on('error', err => reject((err));
resSearch.on('end', result => {
if (result.status === 0 && entries.length === 1) {
return resolve({
cn: entries[0].cn,
objectclass: entries[0].objectclass,
password: entries[0].password
});
} else {
return reject(new Error(`Wrong search result: ${result}`));
}
});
});
});
});
}
I am using mockery and Sinon to replace ldapjs dependency inside my module:
beforeEach(function () {
searchEM = new EventEmitter();
sandbox = sinon.createSandbox();
ldapClientStub = Stubs.getLdapClientStub(sandbox);
ldapClientStub.bind.yields(null);
ldapClientStub.search.withArgs('o=ldap', {filter: `(cn=${findParam})`}).yields(null, searchEM);
mockery.registerMock('ldapjs', Stubs.getLdapStub(ldapClientStub));
mockery.registerAllowable('../src/client');
UserClientCls = require('../src/client').default;
userClient = new UserClientCls(config.get());
});
it('should return user with given username', function (done) {
setTimeout(() => {
searchEM.emit('searchEntry', users[1]);
searchEM.emit('end', {status: 0});
console.log('emitted');
}, 500);
searchEM.on('end', res => console.log(res));
userClient.findUser(findParam)
.then(user => {
user.cn.should.equal(users[1].attributes.cn);
user.objectclass.should.equal(users[1].attributes.objectclass);
user.password.should.equal(users[1].attributes.password);
return done();
})
.catch(err => done(err));
});
The problem is that listeners defined inside findUser are never called (but the function itself is called). The listener I defined in the test (just to debug the behaviour) is correctly called.
I do not understand if I miss something about how EventEmitters works or if I am doing the test in a wrong way. Or maybe I wrote a bad piece of code that cannot be tested.
I found a solution to my problem. I extended the base EventEmitter: I added the logic to store which event I want to emit and overrode its on method with a logic to emit my fake event.
class TestEventEmitter extends EventEmitter {
constructor() {
super();
}
setFakeEmit(fakeEmit) {
this.fakeEmit = fakeEmit;
}
on(eventName, cb) {
super.on(eventName, cb);
if (super.eventNames().length === 4)
this.fakeEmit.forEach(f => this.emit(f.name, f.obj));
}
}
So, in beforeEach I can stub ldapClientStub.search to make it return my TestEventEmitter:
beforeEach(function() {
searchEM = new TestEventEmitter();
searchEM.setFakeEmit([{
name: 'searchEntry',
obj: { object: users[1].attributes }
}, {
name: 'end',
obj: { status: 0 }
}]);
...
ldapClientStub.search.withArgs('o=ldap', { filter: `(&(cn=${findParam})(objectclass=astsUser))` }).yields(null, searchEM);
})
This solution may be not very elegant, but it works. If someone can post a better solution I'll be glad to have a look.