Node JS Puppeteer throws error at url with proxy - node.js

Question

I'm making a automation script for filling a form online with puppeteer, and to not blacklist ip's I decided to use proxies for each request, this is the part which gives me error
console.log(`profile ${ii} started`)
let proxy = await proxy_rotation(proxy_data, ii);
console.log("using proxy: ", proxy);
let exec_path = path.resolve(path.dirname(process.execPath) + "/node_modules/puppeteer/.local-chromium/win64-869685/chrome-win/chrome.exe")
const browser = await puppeteer.launch({
executablePath: exec_path,
args: ['--disable-web-security']
});
const page = await browser.newPage();
console.log("1");
await page.setRequestInterception(true);
await useProxy(page, proxy);
console.log("2");
await page.goto(data[ii][0]); //this is where the error gets thrown
this part below doesn't get to run when using a proxy, without it, runs smotthly
console.log("3");
await page.type("#name", data[ii][1]);
await page.type("#yourEmail", data[ii][2]);
await page.type("#phone", data[ii][3]);
await page.type("#street", data[ii][4]);
await page.type("#city", data[ii][5]);
await page.type("#psc", data[ii][6]);
await page.select('select#state', data[ii][7]);
await page.select('select#prefered_size_sel', data[ii][8]);
await page.$eval('input[name="agreed_personal_info_tiny_contact_form"]', check => check.checked = true);
await page.evaluate(() => {
document.querySelector('input[name="agreed_personal_info_tiny_contact_form"]').click();
});
I just console logged a few numbers, to debug where the script is getting stuck. I also tested the proxy and website I'm trying to access both with a proxy tester and manually, and had no problem accessing it
but when I run my script I get this
I understand it pretty much says it cannot access the url, but there should be no reason for that. Do I need to change the way I'm acessing the url when using a proxy? Or add some extra args to the browser? Can I get a more specific error message somehow? Thanks for any suggestions
Also this is the puppeteer function that throws the error
async function navigate(client, url, referrer, frameId) {
try {
const response = await client.send('Page.navigate', {
url,
referrer,
frameId,
});
ensureNewDocumentNavigation = !!response.loaderId;
return response.errorText
? new Error(`${response.errorText} at ${url}`)
: null;
}
catch (error) {
return error;
}
}

Answer 1

That error indicates that something is off how you are using your proxy. Is useProxy your own function or the one from puppeteer-page-proxy? You mention setting the proxy per-request but seems you are setting it for the whole page, is that intentional? The way your proxy is formatted also seems off- check how I do it below.
You can try launching the browser with your proxy server and using page.authenticate() to handle auth. Like this:
let proxy = await proxy_rotation(proxy_data, ii);
const [host, port, username, password] = proxy.split(':');
const parsedProxy = new URL(`http://${username}:${password}#${host}:${port}`);
const browser = await puppeteer.launch({
executablePath: exec_path,
args: ['--disable-web-security', '--ignore-certificate-errors', `--proxy-server=${parsedProxy.host}`]
});
const page = await browser.newPage();
await page.authenticate({
username: parsedProxy.username,
password: parsedProxy.password,
});
Before doing that I would change what you pass to useProxy such that it looks like http://username:pw#host:port (Lines 2-3).