I am trying to go to a web page click login which will direct me to another website(which is Steam) to type username/password in. but when I click to direct me to steam it just hangs there.
my code:
try {
await page.goto("https://www.tradeit.gg", {waitUntil: "load", timeout: 0});
} catch(e) {
console.log(e)
}
try{
await Promise.all([
page.waitForSelector('a[href="https://steamtrade.gg/auth/steam"]'),
page.click('a[href="https://steamtrade.gg/auth/steam"]'),
page.waitForNavigation({ waitUntil: 'networkidle0' })
])
} catch (e) {
console.log("can not login, error: " + e);
}
the button I tried to click:
<ul class="navbar-nav mr-3" style="color: #c7ced4;">
<a href="https://steamtrade.gg/auth/steam" class="mr-3" onclick="if (!window.__cfRLUnblockHandlers) return false; fireLoginEvent('navbar_mainpage')">
<button class="btn btn-success steamlogin my-2 my-sm-0" type="submit"></button>
</a>
</ul>
the error I got:
can not login, error: TimeoutError: Navigation timeout of 30000 ms exceeded
(node:30861) UnhandledPromiseRejectionWarning: Error: No node found for selector: input[type="password"]
at Object.exports.assert (/home/danhle/Documents/Work/personalProjects/googleAdd/scrawlingApp/node_modules/puppeteer/lib/cjs/puppeteer/common/assert.js:26:15)
at DOMWorld.type (/home/danhle/Documents/Work/personalProjects/googleAdd/scrawlingApp/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:306:21)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async /home/danhle/Documents/Work/personalProjects/googleAdd/scrawlingApp/app.js:31:5
(node:30861) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
I really need help its been days ...
To me, you make too complicated, try to keep it as simple as possible.
This code works just fine:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://www.tradeit.gg', { waitUntil: 'networkidle2' });
await Promise.all([
page.waitForNavigation({ waitUntil: 'networkidle2' }),
page.click('[href="https://steamtrade.gg/auth/steam"]')
]);
await browser.close();
})();
Even your code works if you change
await page.goto("https://www.tradeit.gg", {waitUntil: "load", timeout: 0});
in a way where you don't wait for the load event.
Related
I'm new to using node.js and puppeteer and i keep getting three errors and i can't seem to work out why i'm getting the errors or how i can fix them.
I'm trying to scrape the same website over and over again but sometimes end up getting the errors.
Error one:
UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'close' of undefined
Error two:
UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
Error three:
DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
My code:
const puppeteer = require("puppeteer");
const referers = require('./models/referers.json');
const agents = require('./models/agents.json');
const log = console.log;
const waitForDelay = (time) => {
try {
return new Promise((resolve) => {
setTimeout(resolve, time);
});
} catch (error) {
log('delay error', error);
}
};
const runPuppeteer = async() => {
let browser, page;
try {
log('started puppeteer');
const randomAgent = agents[Math.floor(Math.random() * agents.length)];
const randomReferer = referers[Math.floor(Math.random() * referers.length)];
log('randomAgent', randomAgent, '\n randomReferer', randomReferer);
browser = await puppeteer.launch({
headless: true,
ignoreHTTPSErrors: true,
slowMo: 10,
args: [
"--no-sandbox",
"--disable-setuid-sandbox"
]
});
page = await browser.newPage();
await page.setUserAgent(randomAgent);
await page.setJavaScriptEnabled(true);
await page.setExtraHTTPHeaders({ referer: randomReferer, 'Accept-Language': 'en' });
await page.setViewport({ width: 1680, height: 1050 });
await page.goto('https://SiteUrl.com', { waitUntil: 'domcontentloaded', timeout: 60000 });
await page.waitForSelector('.mwButton', { visible: true, timeout: 60000 });
const buttonElement = await page.$('.mwButton');
if (await buttonElement.isIntersectingViewport()) {
await page.click(".mwButton");
log('button clicked');
log(await page.title());
await waitForDelay(10000);
await page.screenshot({ path: 'screenshot.png' });
} else {
log('button not clicked');
}
} catch(error) {
log(error.message);
} finally {
await browser.close();
log('closing browser');
await setTimeout(runPuppeteer, 10000);
}
};
Those are very common and generic errors node throws when you have an exception on an async function.
The problem in your code is the finally clause, it attempts to close the browser, but when your code fails to create a browser object, it remains undefined when it reaches finally.
You could do if(browser) await browser.close() to make sure it exists, if not there is no point in trying to close it...
i have create test.js file, and installed one npm library, when i run the file i am getting below error, i tried to use npm i puppeteer this library but i am getting error when i tried to install it, so i used this library npm i puppeteer-core
(node:30075) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:30075) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
here is my full code for test.js file
const puppeteer = require('puppeteer-core');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://example.com');
await page.screenshot({path: 'example.png'});
await browser.close();
})();
can anyone please help me why i am getting this error ?
When using async/await, it is recommended to use try/catch catch the error, and to figure out what's wrong.
(async () => {
try {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://example.com');
await page.screenshot({path: 'example.png'});
await browser.close();
} catch (error) {
console.log('error', error);
// Do whatever you want, throw the error again if you want but it will just produce `UnhandledPromiseRejectionWarning` again, if you throw it again.
}
})();
Running puppeteer with Proxy authentication failing.
Puppeteer version: 1.8
Platform / OS version:MacOS 10.13.6
Node.js version:v10.9.0
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
headless:false,
ignoreHTTPSErrors:true,
devtools:true,
timeout:3000,
args: ['--no-sandbox','--proxy-server=xxx:xxx']
});
const user='xxx';
const password='xxx';
const page = await browser.newPage();
// await page.setExtraHTTPHeaders({
// 'Proxy-Authorization': 'Basic ' + Buffer.from(`${user}:${password}`).toString('base64'),
// });
await page.authenticate({username:user, password:password});
await page.goto('https://www.apple.com/');
let title = await page.title();
console.log("title:" + title);
await browser.close();
})();
got error:
(node:5858) UnhandledPromiseRejectionWarning:
Error: net::ERR_TUNNEL_CONNECTION_FAILED at https://www.apple.com/
at navigate (/xxx/node_modules/_puppeteer#1.8.0#puppeteer/lib/Page.js:622:37)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:5858) UnhandledPromiseRejectionWarning:
Unhandled promise rejection. This error originated either by throwing
inside of an async function without a catch block,
or by rejecting a promise which was not handled with .catch().
(rejection id: 1)
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
headless:false,
ignoreHTTPSErrors:true,
devtools:true,
timeout:3000,
args: ['--no-sandbox','--proxy-server=https=xxx:xxx']
});
const user='xxx';
const password='xxx';
const page = await browser.newPage();
await page.authenticate({username:user, password:password});
await page.goto('https://www.apple.com/');
let title = await page.title();
console.log("title:" + title);
await browser.close();
})();
For me it worked by adding https between the proxy server argument.
Within your args to launch puppeteer,
You can also try this format:
--proxy-server=http=xxx:xxxx
OR
Have you tried setting proxy like this '--proxy-server=https=MyWorkingProxyIP:Port' according to docs https://www.chromium.org/developers/design-documents/network-settings ?
I have this code:
// <-- add event on top of file
process.on("unhandledRejection", (reason, p) => {
console.error("Unhandled Rejection at: Promise", p, "reason:", reason);
// browser.close(); // <-- no need to close the browser here
});
const puppeteer = require('puppeteer');
async function getPic() {
try{ // <-- wrap the whole block in try catch
const browser = await puppeteer.launch(/*{headless: false}*/);
const page = await browser.newPage();
await page.setViewport({width: 1000, height: 500}); // <-- add await here so it sets viewport after it creates the page
//await page.goto('https://www.google.com'); //Old way of doing. It doesn't work for some reason...
page.goto('https://www.google.com/'); // async
// wait for either of events to trigger
await Promise.race([
page.waitForNavigation({waitUntil: 'domcontentloaded'}),
page.waitForNavigation({waitUntil: 'load'})
]);
await page.screenshot({path: 'pic.png'});
await browser.close(); // <-- close browser after everything is done
} catch (error) {
console.log(error);
}
}
getPic();
And i get this error:
Unhandled Rejection at: Promise Promise {
<rejected> Error: Navigation Timeout Exceeded: 30000ms exceeded
at Promise.then (C:\...\pupet test\node_modules\pupp
eteer\lib\NavigatorWatcher.js:71:21)
at <anonymous> } reason: Error: Navigation Timeout Exceeded: 30000ms exceede
d
at Promise.then (C:\...\pupet test\node_modules\pupp
eteer\lib\NavigatorWatcher.js:71:21)
at <anonymous>
I read that the 'unhandledRejection' event is emitted whenever a Promise is rejected and no error handler is attached to the promise within a turn of the event loop.
So from this particular error message, i cannot undedrstand what is causing the problem.
So how i can figure out which is the erroneous promise so i can further understand what went wrong and mitigate it?
I am trying to take a screenshot with puppeteer.
I had many troubles with Unhandled Promise Rejection errors.
I browsed many forums and topics that suggested many different things, which i applied.
These suggestions ranged from using async function, to using try-catch blocks.
After improvements, this is my final version of the code:
const puppeteer = require('puppeteer');
async function getPic() {
const browser = await puppeteer.launch(/*{headless: false}*/);
const page = await browser.newPage();
page.setViewport({width: 1000, height: 500});
process.on("unhandledRejection", (reason, p) => {
console.error("Unhandled Rejection at: Promise", p, "reason:", reason);
browser.close();
});
try{
await page.goto('https://www.google.com');
}
catch (error) {
console.log(error);
browser.close();
}
await page.screenshot({path: 'pic.png'});
await broswer.close();
}
getPic();
Though, still it does not work and i get this error message:
Error: Navigation Timeout Exceeded: 30000ms exceeded
at Promise.then (C:\...\pupet test\node_modules\pupp
eteer\lib\NavigatorWatcher.js:71:21)
at <anonymous>
Unhandled Rejection at: Promise Promise {
<rejected> { Error: Protocol error (Target.sendMessageToTarget): Target closed
.
at Promise (C:\...\pupet test\node_modules\puppeteer
\lib\Connection.js:198:56)
at new Promise (<anonymous>)
at CDPSession.send (C:\...\pupet test\node_modules\p
uppeteer\lib\Connection.js:197:12)
at Page._screenshotTask (C:\...\pupet test\node_modu
les\puppeteer\lib\Page.js:707:24)
at <anonymous>
message: 'Protocol error (Target.sendMessageToTarget): Target closed.' } } rea
son: { Error: Protocol error (Target.sendMessageToTarget): Target closed.
at Promise (C:\...\pupet test\node_modules\puppeteer
\lib\Connection.js:198:56)
at new Promise (<anonymous>)
at CDPSession.send (C:\...\pupet test\node_modules\p
uppeteer\lib\Connection.js:197:12)
at Page._screenshotTask (C:\...\pupet test\node_modu
les\puppeteer\lib\Page.js:707:24)
at <anonymous>
message: 'Protocol error (Target.sendMessageToTarget): Target closed.' }
Use async await properly
You need to use async-await properly. You are calling asynchronous functions in synchronous way.
Here is the modified code with proper await. Read the code below and see what's wrong.
const puppeteer = require('puppeteer');
async function getPic() {
try{ // <-- wrap the whole block in try catch
const browser = await puppeteer.launch(/*{headless: false}*/);
const page = await browser.newPage();
await page.setViewport({width: 1000, height: 500}); // <-- add await here so it sets viewport after it creates the page
await page.goto('https://www.google.com');
await page.screenshot({path: 'pic.png'});
await broswer.close(); // <-- close browser after everything is done
} catch (error) {
console.log(error);
}
}
getPic();
Check proxy settings
The error says it is timing out, which means it cannot even load the website. Try to check if the page loads properly on the browser.