Puppeteer: return JSON response of AJAX response - node.js

Question

while the page is loading I am trying to wait for a certain AJAX request made by my page and then return its response's JSON body. My code does not stop iterating through every response even after the condition is met within the listener for 'response' event.
Once I find the response I want to return, how can I capture the JSON from the response, stop execution the page from loading further, and return my JSON?
async function runScrape() {
const browser = await browserPromise;
const context = await browser.createIncognitoBrowserContext();
const page = await context.newPage();
await page.setDefaultTimeout(60000);
let apiResponse;
page.on('response', async response => {
let url = await response.url();
let status = await response.status();
console.info(status + " NETWORK CALL: " + url);
if ( url.match(requestPattern) ) {
apiResponse = await response.text();
await page.evaluate(() => window.stop());
}
});
await page.goto(req.query.url);
console.log("API RESPONSE:\n" + apiResponse);
return apiResponse
}}
=== UPDATE ===
This was the solution that ended up working. It seemed this approach was required due to the specific behavior of the page being scraped.
async function runScrape() {
const browser = await browserPromise;
const context = await browser.createIncognitoBrowserContext();
const page = await context.newPage();
await page.setDefaultTimeout(60000);
await page.setRequestInterception(true);
let JSONResponse;
page.on('response', async response => {
if ( !JSONResponse && response.url().match(requestPattern) ) {
JSONResponse = await response.text();
}
});
page.on('request', request => {
if (request.resourceType() === 'image' || request.resourceType() === 'stylesheet') request.abort()
else request.continue()
});
await page.goto(scrapeURL, {waitUntil: 'networkidle2'});
await page.close();
return JSONResponse
}
runScrape()
.then( response => {
res.setHeader("content-type", "application/json");
res.status(200).send(response);
})
.catch(err => {
let payload = {"errorType": err.name, "errorMessage": err.message+"\n"+err.stack};
console.error(JSON.stringify(payload));
res.status(500).json(payload);
});

Answer 1

I would simplify it to a single page.on('response'... where we are looking for the desired request pattern with String.includes().
Once the response is identified then we can emulate the "Stop loading this page" button of the browser with await page.evaluate(() => window.stop()). The window.stop() method won't close the browser yet, just stops the network requests.
let resp
page.on('response', async response => {
if (response.url().includes(requestPattern)) {
resp = await response.json()
await page.evaluate(() => window.stop())
}
})
await page.goto(req.query.url, { waitUntil: 'networkidle0' } )
console.log(resp)
Edit:
To avoid undefined response you should use waitUntil: 'networkidle0' setting on page.goto(), see the docs about the options. You've got undefined because by default puppeteer considered page to be loaded when the load event is fired on the page (this is the default setting of waitUntil). So if the page considered loaded but there are still network connections in the queue and your request pattern is not found yet: the script will go on from goto to console.log. So you make sure the request is registered before it would happen by waiting until all network request has been finished.
networkidle0: consider navigation to be finished when there are no more than 0 network connections for at least 500 ms.
Please note: by setting networkidle you won't be able to disconnect after the request pattern condition was fulfilled, so your plan to stop the responses won't be possible.
I recommend to abort those resourceTypes which are not needed, like this you may have similar results as you would with stopping the requests:
For example:
Place it right after the page.on('response', async response => {... block ended.
await page.setRequestInterception(true)
page.on('request', request => {
if (request.resourceType() === 'image' || request.resourceType() === 'stylesheet') request.abort()
else request.continue()
})
You can use it with a request.url().includes(unwantedRequestPattern) condition as well if you know which connections you don't need.