I am using puppeteer to automate a website, puppeteer session may take about (30s-60s), and I want to fire a request(open another puppeteer session) right after the one before it finishes and I realized that I can't use setInterval because time is not constant in my case, how can I achieve such thing
Use a recursive function to keep calling the same function right after it's done.
async function bot(){
const browser = await puppeteer.launch()
// other code
await browser.close()
}
async function fireMyRequest(){
await bot()
await fireMyRequest()
}
Related
I've a main function that call other 2 functions with await.
The first one will download an image locally,
the second one will upload an image on our CDN with an Axios call, and wait his JSON response.
But when the first one will complete the main function going out, and not join on the second.
I need to join it
Note: Anyway the console.log('Middle') still never printed!
async function main(){
console.log('[1] Joined in the main()');
let url_immagine = 'https://www.castrol.com/content/dam/castrol/country-sites/en_gb/united-kingdom/home/motorcycle-engine-oils/engine-oil-brands/power1-racing-4t-10w-40.png.img.500.medium.png';
let filename = 'motor_oil.png';
let id = 122;
await downloadImagefromRemote(url_immagine, filename);
console.log('middle');
await putImageOnCloud(id, filename);
}
this is my JSFiddle script:
Code
You could just not await the second function.
putImageOnCloud(id, filename)
You say you dont want to wait for it, but you still use await.
I have a code that looks like the following
console.time('setfoo')
redis.set('foo', 'bar')
...
let result = await redis.get('foo')
console.timeEnd('setfoo')
Response time for this get is 263ms but when I remove await it goes down to 0.037ms What could be the reason?
Using await basically means "wait for this instruction to be completed before you do anything else".
Not using it would make your script execute quickly, but the result variable will only contain an unresolved Promise. More on promises here
In this case, redis does not use Promises natively, await is not needed.
What I don't get is your bit of code here:
let result = await redis.get(
redis.get('foo')
)
You pass a call to redis as a parameter to another call to redis. Redis natively uses callbacks, so if you want to display the variable you got from redis, try:
let result = await redis.get('foo', (result) => { console.log(result) })
Also, since redis.set() is asynchronous, by the time you try to recover your data, 'foo' might not have been set yet.
This might help you use promises with redis
EDIT: Your code re-written according to the redis documentation
const { promisify } = require("util");
const getAsync = promisify(client.get).bind(redis);
const setAsync = promisify(client.set).bind(redis);
...
console.time('setFoo')
await setAsync('foo', 'bar')
...
let result = await getAsync('foo', 'bar')
console.timeEnd('setfoo')
Just to throw my 2 cents here: V8 (the JavaScript engine behind browsers and NodeJS) is using an event-loop (powered by libuv) which does not necessarily execute tasks on a separate thread. What this event loop does is essentially accepting jobs and executing them when it can. More info on the event loop here.
With that being said, .get call adds a task to the event loop, which translates into a JavaScript Promise object. If you do not await that, the task is just added to the event loop. From your example:
console.time('foo')
let result = redis.get('foo')
console.timeEnd('foo')
What you are actually measuring here is how much time does it take for the event loop task to be submitted?, which is insignificant.
Awaiting on the .get call will not only add the task to the event loop, but also block until the task executes:
console.time('foo')
let result = await redis.get('foo')
console.timeEnd('foo')
So, to answer your question: the two times should not be equal at all, because one example is measuring how long it takes for Redis to process the request and return the result, while the other example is measuring just how long it takes for the job to be submitted to the event loop.
Im using testcafe for GUI tests. I'm running the node server in the background and then start testing. When I'm navigating though the GUI with testcafe API it all works great, but when I'm trying to call a function that changes the db (hard coded in the test) the database isn't effected at all.
Here is my code:
fixture('Permissions')
.page('https://localhost');
test('go to permissions', async browser => {
await onlineFormsController.createOnlineFrom("OMRI",'NEWFORM',async ()=>{
await browser.click('#editUsersTree');
await browser.click('#loadDefaultTree');
await browser.wait(500);
await browser.pressKey('enter');
await browser.wait(500);
await browser.pressKey('enter');
await browser.click('#saveTree');
await browser.pressKey('enter');
await browser.navigateTo('https://localhost/Home');
await browser.wait(5000);
});
});
The function onlineFormsController.createOnlineFrom should create new form in my database, but nothing happened. It's working good separately form the testcafe test (tested it with mocha and it works great), but when I'm running the test with testcafe it's like this line is ignored or have no effects at all.
Any ideas what causing this problem?
TestCafe should not interfere with MongoDB work.
I recommend you the following:
ensure that the createOnlineFrom returns a Promise object;
do not pass function with test actions as a parameter. Since the createOnlineFrom returns a Promise object, you can organize your code in a more readable way:
test('go to permissions', async browser => {
await onlineFormsController.createOnlineFrom("OMRI",'NEWFORM');
await browser.click('#editUsersTree');
});
If these recommendations do not help, please provide the full code of your createOnlineFrom function.
How come waitForFunction, waitForSelector, await page.evaluate etc. all give errors UNLESS I put a 10 seconds delay after reading the page?
I would think these were made to wait for something to happen on the page, but without my 10 seconds delay (just after page.goto) - all of them fail with errors.
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto('https://sunnythailand.com')
console.log("Waiting 10 seconds")
await new Promise((resolve)=>setTimeout(()=> resolve() ,10000));
console.log("Here we go....")
console.log("waitForFunction START")
await page.waitForFunction('document.querySelector(".scrapebot_description").textContent.length > 0');
console.log("waitForFunction FOUND scrapebot")
console.log("Waiting for evaluate")
const name = await page.evaluate(() => document.querySelector('.scrapebot_description').textContent)
console.log("Evaluate: " + name)
await browser.close()
})()
My theory is that our sunnythailand.com page sends an "end of page" or something BEFORE it finished rendering, and then all the waitFor functions go crazy and fail with all kinds of strange errors.
So I guess my question is... how do we get waitFor to actually WAIT for the event to happen or class to appear etc...?
Don't use time out cause you don't know how much time for it will take to load full page. It depends person to person on his internet bandwidth.
All you need to rely on promise for your class
await page.waitForSelector('.scrapebot_description');
lets wait for your particular class then it will work fine
Please remove this
//await new Promise((resolve)=>setTimeout(()=> resolve() ,5000));
plese let me know your test result after this. I am sure it will solve.
I have just started with async and await and trying to convert all my callback syntax to async/await style.
One thing I could not understand is, why I need to every time prefix my function with async keyword.
Here is the example:
APIQuery.js
makeRequest: async(options) => {
try {
var response = await(request(options1))
}
catch(err){
console.log("Some error occurred");
response = undefined;
}
return response;
}
MobileAPI.js
getMobileData: async modal => {
var options = {method: 'GET', json: true,uri: 'https://example.com/mobile/'+modal}
var response = await APIQuery.makeRequest(options);
}
MobileService.js
getMobileDataService: async modal => {
var response = await MobileAPI.getMobileData(modal);
}
MobileController.js
Similarly again I have to use async and await combination to return response.
So my question is, is there a way to get rid of using this everywhere. Calling async await inside APIQuery.js is not enough?
If you want to use the await operator, you have to use async keyword before function declaration:
The await operator is used to wait for a Promise. It can only be used inside an async function.
If you don't want to use async everywhere, you can continue using callbacks or Promises (then, catch, etc.).
So my question is, is there a way to get rid of using this everywhere.
You can't block. Period. As soon as you block, your server would become completely unresponsive to all requests until it's unblocked. For this reason, your code must be asynchronous.
Callbacks are a perfectly valid form of asynchrony, but async/await is easier to use, understand, and maintain.