I'm currently setting up a CI environment to automate e2e tests our team runs in a test harness. I am setting this up on Gitlab and currently using Puppeteer. I have an event that fires from our test harness that designates when the test is complete. Now I am trying to "pool" the execution so I don't use up all resources or run out of listeners. I decided to try out "puppeteer-cluster" for this task. I am close to having things working, however I can't seem to get it to wait for the event on page before closing the browser. Prior to using puppeteer-cluster, I was passing in a callback to my function and when the custom event was fired (injected via exposeFunction), I would go about calling it. That callback function is now being passed in data though now and therefore not waiting. I can't seem to find a way to get the execution to wait and was hoping someone might have an idea here. If anyone has any recommendations, I'd love to hear them.
test('Should launch the browser and run e2e tests', async (done) => {
try {
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
maxConcurrency: 10,
monitor: false,
timeout: 1200000,
puppeteerOptions: browserConfig
});
// Print errors to console
cluster.on("taskerror", (err, data) => {
console.log(`Error crawling ${data}: ${err.message}`);
});
//Setup our task to be run
await cluster.task( async ({page, data: {testUrl, isLastIndex, cb}, worker}) => {
console.log(`Test starting at url: ${testUrl} - isLastIndex: ${isLastIndex}`);
await page.goto(testUrl);
await page.waitForSelector('#testHarness');
await page.exposeFunction('onCustomEvent', async (e) => {
if (isLastIndex === true){ ;
//Make a call to our callback, finalizing tests are complete
cb();
}
console.log(`Completed test at url: ${testUrl}`);
});
await page.evaluate(() => {
document.addEventListener('TEST_COMPLETE', (e) => {
window.onCustomEvent('TEST_COMPLETE');
console.log("TEST COMPLETE");
});
});
});
//Perform the assignment of all of our xml tests to an array
let arrOfTests = await buildTestArray();
const arrOfTestsLen = arrOfTests.length;
for( let i=0; i < arrOfTestsLen; ++i){
//push our tests on task queue
await cluster.queue( {testUrl: arrOfTests[i], isLastIndex: (i === arrOfTestsLen - 1), cb: done });
};
await cluster.idle();
await cluster.close();
} catch (error) {
console.log('ERROR:',error);
done();
throw error;
}
});
So I got something working, but it really feels hacky to me and I'm not really sure it is the right approach. So should anyone have the proper way of doing this or a more recommended way, don't hesitate to respond. I am posting here shoudl anyone else deal with something similar. I was able to get this working with a bool and setInterval. I have pasted working result below.
await cluster.task( async ({page, data: {testUrl, isLastIndex, cb}, worker}) => {
let complete = false;
console.log(`Test starting at url: ${testUrl} - isLastIndex: ${isLastIndex}`);
await page.goto(testUrl)
await page.waitForSelector('#testHarness');
await page.focus('#testHarness');
await page.exposeFunction('onCustomEvent', async (e) => {
console.log("Custom event fired");
if (isLastIndex === true){ ;
//Make a call to our callback, finalizing tests are complete
cb();
complete = true;
//console.log(`VAL IS ${complete}`);
}
console.log(`Completed test at url: ${testUrl}`);
});
//This will run on the actual page itself. So setup an event listener for
//the TEST_COMPLETE event sent from the test harness itself
await page.evaluate(() => {
document.addEventListener('TEST_COMPLETE', (e) => {
window.onCustomEvent('TEST_COMPLETE');
});
});
await new Promise(resolve => {
try {
let timerId = setInterval(()=>{
if (complete === true){
resolve();
clearInterval(timerId);
}
}, 1000);
} catch (e) {
console.log('ERROR ', e);
}
});
});
Related
The code below is an example of what may take place during development.
With the current code, the outer function may throw an error but in this case wont. However, the nested function WILL throw an error (for examples sake). Once it throws the error it cannot be caught as it is asynchronous function.
Bungie.Get('/Platform/Destiny2/Manifest/').then((ResponseText)=>{
//Async function that WILL throw an error
Bungie.Get('/Platform/Destiny2/Mnifest/').then((ResponseText)=>{
console.log('Success')
})
}).catch((error)=>{
//Catch all errors from either the main function or the nested function
doSomethingWithError(error)
});
What I want is for the outer most function to catch all asynchronous function error's but with this code I cannot. I have tried awaiting the nested function but there may be certain circumstances where it will be quicker to not wait for the function. I also tried to include a .catch() with each nested function but this would require a .catch() for each function that would allhandle the error in the same way e.g. doSomethingWithError().
you only needs return the inner function in the outside function.
see example below:
const foo = new Promise((resolve,reject) =>{
setTimeout(() => resolve('foo'), 1000);
});
foo.then((res)=>{
console.log(res)
return new Promise((resolve,reject)=>{
setTimeout(() => reject("bar fail"), 1000);
})
}).catch((e)=>{
// your own logic
console.error(e)
});
this is called promise chaining. see this post for more info https://javascript.info/promise-chaining
if you have multiple promises can do something like:
const foo1 = new Promise((resolve,reject) =>{
setTimeout(() => resolve('foo1'), 1000);
});
const foo2 = new Promise((resolve,reject) =>{
setTimeout(() => resolve('foo2'), 2000);
});
const foo3 = new Promise((resolve,reject) =>{
setTimeout(() => reject('foo3'), 3000);
});
const bar = new Promise((resolve,reject) =>{
setTimeout(() => resolve('bar'), 4000);
});
foo1
.then((res)=>{
console.log(res)
return foo2
})
.then((res)=>{
console.log(res)
return foo3 // throws the error
})
.then((res)=>{
console.log(res)
return bar
})
.catch((e)=>{
// every error will be cached here
console.error(e)
});
I would aim to use async / await unless you have very particular reasons, since it avoids callback hell and makes your code simpler and more bug free.
try {
const response1 = await Bungie.Get('/Platform/Destiny2/Manifest/');
const response2 = await Bungie.Get('/Platform/Destiny2/Mnifest/');
console.log('Success');
} catch (error) {
doSomethingWithError(error);
}
Imagine each Bungie call takes 250 milliseconds. While this is occurring, NodeJS will continue to execute other code via its event loop - eg requests from other clients. Awaiting is not the same as hanging the app.
Similarly, this type of code is used in many browser or mobile apps, and they remain responsive to the end user during I/O. I use the async await programming model in all languages these days (Javascript, Java, C#, Swift etc).
Try this:
let getMultiple = function(callback, ... keys){
let result = [];
let ctr = keys.length;
for(let i=0;i<ctr;i++)
result.push(0);
let ctr2 = 0;
keys.forEach(function(key){
let ctr3=ctr2++;
try{
Bungie.Get(key, function(data){
result[ctr3] = data;
ctr--;
if(ctr==0)
{
callback(result);
}
});
} catch(err) {
result[ctr3]=err.message;
ctr--;
if(ctr==0)
{
callback(result);
}
}
});
};
This should get all your data requests and replace relevant data with error message if it happens.
getMultiple(function(results){
console.log(results);
}, string1, string2, string3);
If the error causes by requesting same thing twice asynchronously, then you can add an asynchronous caching layer before this request.
I'm trying to download videos from some URLs and for some reason the downloads will sometimes stop before they're complete - as in res.on('data', ...) will stop being called (using https.get) before the response has completed. I would like to be able to detect this and retry the request after a given time, but I'm having a hard time getting working code for it. I'm hoping there would be a built in way to deal with something like this that I'm missing. I've already tried setting a timeout for the request and setting a timeout for the response and neither of those seem to do the trick.
Current attempt:
async function downloadVideo(send) {
try {
https.get(downloadUrl, res => {
let timerId;
res.on(`data`, (c) => {
clearTimeout(timerId);
timerId = setTimeout(() => {
res.destroy();
console.log(`retrying`, name);
downloadVideo(send);
}, 2000);
if (!throttle) {
send(`PercentDownloaded`, [
index,
100 * fileSize(filePath) / res.headers[`content-length`]
]);
throttle = true;
setTimeout(() => throttle = false, 500);
}
});
res
.pipe(fs.createWriteStream(filePath))
.on(`finish`, () => {
clearTimeout(timerId);
send(`Done`, index);
});
});
} catch (error) {
console.log(name, error);
send(`DownloadError`, index);
}
}
I am trying to build a loop that my server will run, pinging an arduino which will return data when it receives a packet from my IP. I already have the mongoDB collection working for the data and am just stuck at this point where I would like my server to begin pinging the arduino every few seconds from start up.
Note: this is really my first crack at really understanding async JS and is likely just a noob mistake.
Code:
const dgram = require('dgram');
const message = Buffer.from('Some bytes');
const clientPing = dgram.createSocket('udp4');
const pinging = true;
function ping() {
return new Promise ((resolve, reject) => {
// This chunk
clientPing.send("Hi Uno", 8888, 'xxx.xx.xx.xxx', (err) => {
clientPing.close();
});
// When run on it's own successfully pings the arduino
const error = false;
if(error == true) {
reject('Error');
} else {
resolve('Pinged');
}
});
}
//Working ping
ping().then().catch(err => console.log(err));
All of this above was meant to act as just a simple proof that the promise does work and successfully emits a udp packet to the target.
function loop() {
while(pinging == true) {
setTimeout(() => {
ping().then().catch(err => console.log(err));
console.log('Pinged');
}, 3000);
}
}
The rest of these are just different combinations of how I've tried to solve the problem here. As far as I can tell I don't have a memory leak as the loop runs just fine without increasing resource consumption.
function loop() {
console.log("entered loop")
while(pinging == true) {
setTimeout(() => {
clientPing.send("Hi Uno", 8888, 'xxx.xx.xx.xxx', (err) => {
clientPing.close();
});
console.log('Pinged');
}, 3000);
}
}
This one was intended to run just the code for the ping without using the promise but still doesn't yield the desired result. The console.log is printed above but it doesn't seem to ever trigger the timeout.
loop();
This just runs one of the two loop() functions.
do {
setTimeout(() => {
ping().then().catch(err => console.log(err));
console.log("pinged");
}, 2000)
} while(pinging == true)
Lastly I thought I'd try do but also without success.
If all of these are pasted into a document sequentially it builds my file. Simply uncomment/comment out each section as needed.
Am I just missing something obvious or is there something fairly complex here that's stopping this from working?
Thanks!
Your ping function is written incorrectly. It is resolving promise just after sending ping not on callback of it. Please change it like this.
function ping() {
return new Promise ((resolve, reject) => {
// This chunk
clientPing.send("Hi Uno", 8888, 'xxx.xx.xx.xxx', (err) => {
const error = false;
clientPing.close();
if(error == true) {
reject('Error');
} else {
resolve('Pinged');
}
});
});
}
And for looping if you shouldn't do sync loop it won't wait until setTimeout finished or your function has finished executing.
For async looping you can loop it by async await
for example:
function wait () {
return new Promise((resolve, reject)=> {
setTimeout(() => {
resolve()
}, 3000)
})
}
for (let i = 0; i<n; i++) {
await ping();
await wait();
}
I have the following code that I cannot figure out what I need to do to make this entire process finish before sending a response back to the client. Basically I am looking for the client to receive a response once everything in the method completes or errors out, not before.
I have tried several things, moving things around and adding some .then() and .catch() blocks that may not even be needed. I'm still fairly new to NodeJs so while I understand how things kind of work, I still have not got used to asynchronous coding.
let markStepsComplete = async function() {
let stepsToProcess = new Multimap();//multimap JS npm
let results = await dbUtils.getCompletedSteps();
results.forEach(result => {
stepsToProcess.set(result.ticket_id, result.step_id);
});
return new Promise((resolve,reject)=>{
stepsToProcess.forEachEntry(async function(entry, key) {
let payload = {
ticketId: key,
stepIds: entry
}
let response = await updateStep(payload)//returns promise
if (response.statusCode === 200) {
await Promise.all(payload.stepIds.map(async (stepId) => {
try {
await dbUtils.markStepsCompleted(stepId, payload.ticketId);
} catch(err) {
reject(err);
}
}));
}
else {
await Promise.all(payload.stepIds.map(async (stepId) => {
try {
await dbUtils.markStepsProcessed(stepId, payload.ticketId, response.statusCode);
} catch(err) {
console.log(err);
reject(err);
}
}));
}
});
resolve('Steps Marked Complete');
});//promise end
}
This is the path that i am testing right now that's making the call to the above method.
The updateStep() method is the method that actually calls a HTTP request to an outside REST API. That seems to be working returning a promise.
app.use('/api/posts',async (req,res,next)=>{
let data = await markStepsComplete.markStepsComplete()
.then((data)=>{
res.status(200).json({message: data})
})
.catch((e)=>{
console.log('error from catch:',e);
})
})
The code above runs and does run some stuff in our database as part of this process, but I get back the "Steps Marked Complete" resolve message before the process finishes.
Since the resolve is hit before the process ends, I can never get any of the reject() errors to come back to the client since resolve is called already.
Thanks in advance.
I noticed this snippet and wanted to call it out because it wont work:
payload.stepIds.forEach(async (stepId) => {
try {
await dbUtils.markStepsProcessed(stepId, payload.ticketId, response.statusCode);
} catch(err) {
console.log(err);
reject(err);
}
});
when looping through an array of items that require an async fetch, you can do something like:
await Promise.all(payload.stepIds.map(async (stepId) => {
try {
await dbUtils.markStepsProcessed(stepId, payload.ticketId, response.statusCode);
} catch(err) {
console.log(err);
reject(err);
}
}));
I'm just start with some google cloud services, and I'm trying to get a entity from datastore.
If the client have internet connection, everything its going well.
But i want to put a try catch statement for the cases were the client have no access to datastore, due any reason (like internet).
Here's my code:
try{
let search = datastore.key(['Client', Client_id])
datastore.get(search, /*{timeout: 1000},*/ function (err, entity) {
console.log('limit >>>', entity.limit)
evt.emit('comparedate', res, entity.limit)
});
}
catch(error){
console.log('Error >>>', error)
}
My problem is: there is no time limit for connection attempt. When the client have no access to the internet the request keep "pending" forever, and don't go to the catch condition.
I tried some parameters like: Global#CallOptions, but with no success.
Thanks for any help!
EDIT >>>> I know that's not the most trustworthy way. But for now I resolved with this code:
evt.on('isonline', (res) => {
try{
require('dns').lookup('google.com',function(err) {
if (err && err.code == "ENOTFOUND") {
console.log('NO INTERNET')
evt.emit('readofflinedata', res)
} else {
console.log('WITH INTERNET')
evt.emit('readonlinedata', res)
}
})
}
catch(error){
res.status(200).send({ error: true, message: error.message })
}
})
The Datastore client uses a library internally called google-gax. You can configure timeouts/etc. by passing in gax options.
datastore.get(key, {
gaxOptions: {timeout: 1000}
}, (err, entity) => {
// ...
});
I didn't found any parameter to add a timeout in the get function of datastore. However you can use a Promise and set a timer, if the execution of the function takes too long it will stop it.
var Promise = require("bluebird");
var elt = new Promise((resolve, reject) => {
fun(param, (err) => {
if (err) reject(err);
doSomething(); // <- datastore.get() funtion
resolve();
});
elt.timeout(1000).then(() => console.log('done'))
.catch(Promise.TimeoutError, (e) => console.log("timed out"))