Tried looking through the docs, but didn't find a way to set a max timeout for a test case. Seems like a simple feature.
import puppeteer from 'puppeteer'
test('App loads', async() => {
const browser = await puppeteer.launch({ headless: false, slowMo: 250 });
const page = await browser.newPage();
await page.goto('http://localhost:3000');
await browser.close();
});
Jest's test(name, fn, timeout) function can take a 3rd parameter that specifies a custom timeout.
test('example', async () => {
...
}, 1000); // timeout of 1s (default is 5s)
Source: https://github.com/facebook/jest/issues/5055#issuecomment-350827560
You can also set the timeout globally for a suite using jest.setTimeout(10000); in the beforeAll() function:
beforeAll(async () => {
jest.setTimeout(10000); // change timeout to 10 seconds
});
Related
I have been experiencing this issue from a long time now. I have a web scraper on a Windows VM and I have it set to run every few hours. It works most of the time but a lot of times Puppeteer just opens this page 👇 and not the site or page I want to open.
Why does that happen and what can be the fix for this?
A simple reproduction for this issue can be this code
import puppeteer from 'puppeteer'
import { scheduleJob } from 'node-schedule';
async function run() {
const browser = await puppeteer.launch({
headless: false,
executablePath: chromePath,
defaultViewport: null,
timeout: 0,
args: ['--no-sandbox', '--start-maximized'],
});
const page = await browser.newPage();
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'webdriver', {
get: () => false,
});
});
await page.goto('https://aliexpress.com', {
waitUntil: 'networkidle0',
timeout: 0,
});
}
run();
scheduleJob('scrape aliexpress', `0 */${hours} * * *`, run);
When I run this function on my local machine in the Firebase Emulator it works great, but when I deploy it to the cloud I get a lot of:
TimeoutError: Navigation timeout of 30000 ms exceeded
Code is very simple:
const functions = require("firebase-functions");
const puppeteer = require("puppeteer");
exports.myFunc = functions
.runWith({ memory: '2GB' })
.https.onRequest(async (request, response) => {
const browser = await puppeteer.launch({
headless: true,
args: ['--no-sandbox', '--disable-setuid-sandbox']
});
const page = await browser.newPage();
await page.goto("https://example.com", { waitUntil: 'networkidle2' });
const pageContent = await page.content();
await browser.close();
response.send(pageContent);
});
I get a lot of this in the Firebase Functions log:
You can see this in the Google Cloud report too:
Why are so many runs getting timeout?
Node.js app with Express, deployed on Heroku. It's just dynamic webpages. Loading static webpages works fine.
Loading dynamic webpages works on localhost, but on Heroku it throws me code=H12, desc="Request timeout", service=30000ms, status=503.
In addition, fresh after doing heroku restart or making a deployment, there always seems to be one instance of a status=200 that loads only the static portion of a dynamic webpage.
Screenshot of logs here.
I've tried the following, which have all led to either the same or other unexpected results when deployed on Heroku (such as Error R14 (Memory quota exceeded) and code=H13 desc="Connection closed without response"):
Switching the Puppeteer Heroku buildpack I was using. I've tried the ones mentioned in this troubleshooting guide and this comment.
Adding headless: true in Puppeteer's launch arguments.
Adding the --no-sandbox, --disable-setuid-sandbox, --single-process, and --no-zygote flags in args of Puppeteer's launch arguments. (Reference: this comment & this comment)
Setting the waitUntil argument in Puppeteer's goto function to domcontentloaded, networkidle0 and networkidle2. (Reference: this comment)
Passing a timeout argument in Puppeteer goto function; I've tried 30000 and 60000 specifically, as well as 0 per this comment.
Using the waitForSelector function.
Clearing Heroku's build cache, as per this article.
Printing the url variable (see my code below) in the console. Output is as expected.
I've observed that:
With the code I have right now (see below), the try-catch-finally block never catches any error. It's always one of the following: I get an incomplete result (static portion of requested dynamic webpage), or the app crashes (code=H13 desc="Connection closed without response"). So I haven't been able to get anything out of attempting to print exception in the console from within the catch block.
Any ideas on how I could get this to work?
const app = express();
const puppeteer = require("puppeteer");
let port = process.env.PORT || 3000;
let browser;
...
app.listen(port, async() => {
browser = await puppeteer
.launch({
timeout: 0,
headless: true,
args: [
"--no-sandbox",
"--disable-setuid-sandbox",
"--single-process",
"--no-zygote",
],
});
});
...
app.get("/appropriate-route-name", async (req, res) => {
let url = req.query.url;
let page = await browser.newPage();
try {
await page.goto(url, {
waitUntil: "networkidle2",
});
res.send({ data: await page.content() });
} catch (exception) {
res.send({ data: null });
} finally {
await browser.close();
}
}
Was able to get it to work by using user-agents. Dynamic pages now load just fine on Heroku; requests don't time out every single time anymore.
const app = express();
const puppeteer = require("puppeteer");
let port = process.env.PORT || 3000;
var userAgent = require("user-agents");
...
app.get("/route-name", async (req, res) => {
let url = req.query.url;
let browser = await puppeteer.launch({
args: ["--no-sandbox"],
});
let page = await browser.newPage();
try {
await page.setUserAgent(userAgent.toString()); // added this
await page.goto(url, {
timeout: 30000,
waitUntil: "newtorkidle2", // or "networkidle0", depending on what you need
});
res.send({ data: await page.content() });
} catch (e) {
res.send({ data: null });
} finally {
await browser.close();
}
});
I came across one big issue combining puppeteer with jest. Whenever I hit "npm run test" this test fails displaying: "Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.". This warning also appears even if I pass timeout as a third argument to test function or calling jest.setTimeout(timeout) from inside of beforeEach method callback. What the problem is there, could you guys help me with this. P.S. I'm using jest and puppeteer packages separately
const pup = require('puppeteer')
let browser, page
beforeEach(async _ => {
browser = await pup.launch({
headless: false
})
page = await browser.newPage()
await page.goto('localhost:3000')
})
afterEach(async _ => await browser.close())
test('Login function', async _ => {
await page.click('link')
const url = await page.url()
expect(url).toMatch(/accounts\.google\.com/)
})
In Mocha and Jest it mostly looks the same. You manually must override the default timeout when you run async scripts that will execute longer than the default timeout.
test('Login function', async _ => {
//..
}, 60000);
Alternative you override the global timeout for your test via CLI by using:
--testTimeout=<number>
The following code throws an error, why?
Navigation Timeout Exceeded: 60000ms exceeded
I'm using puppeteer version 1.19.0
const puppeteer = require("puppeteer");
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setCacheEnabled(false);
try {
const response = await page.goto("https://www.gatsbyjs.com", {
waitUntil: "networkidle0",
timeout: 60000
});
console.log("Status code:", response.status());
} catch (error) {
console.log(error.message);
}
await browser.close();
})();
Some other URLs work fine, so I wonder if there is anything special with this particular URL?
If you change the waitUntil : "networkidle2" . There is no time out.
networkidle2 - consider navigation to be finished when there are no
more than 2 network connections for at least 500 ms.
As pointed out in Erez's answer . 'serviceworker' may be holding the connection.You can check it by going to chrome://serviceworker-internals/ . Or Devtools -> Application Tab - Service Wokers
Serive Worker: chrome://serviceworker-internals/
Scope: https://www.gatsbyjs.com/
Registration ID: 295
Navigation preload enabled: false
Navigation preload header length: 4
Active worker:
Installation Status: ACTIVATED
Running Status: RUNNING
Fetch handler existence: EXISTS
Script: https://www.gatsbyjs.com/sw.js
Version ID: 10330
Renderer process ID: 11892
Renderer thread ID: 18124
DevTools agent route ID: 8
From Network : installingWorker ServiceWorker {scriptURL: "https://www.gatsbyjs.com/sw.js", state: "installing", onerror: null, onstatechange: null}
References :
Navigation Timeout Exceeded when using networkidle0 and no insight into what timed out
Support ServiceWorkers #2634
Removing waitUntil: "networkidle0" works so I'm assuming the site is still holding a connection to the server.
I couldn't figure out which connection it is (maybe the service worker?) using the developers tools (accessible in non headless mode by running await puppeteer.launch({ headless: false }))