I am dealing with a request that when called in a browser, will call multiple requests, I managed to capture these requests using puppeteer, but I don't know how to do it in Axios.
for example, make a GET request using Axios to "www.example.com/welcome", this will call about 10 requests inside that request.
in puppeteer, I can do the following to solve this issue:
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', async request => {
const auth = request.headers().authorization;
if (auth) {
TOKEN = auth ? auth.substring(6, auth.length) : TOKEN;
await request.abort();
} else {
await request.continue();
}
});
is there a way to intercept these requests and capture the headers of each of them?
Related
I am working with Next-auth and rtk query. I need that when a request, any, returns a 401 unauthorized error, the page redirects directly to the login. How is it done?
I added 'maxAge: 60' to the [...nextauth].js file and also refetchInterval={30} refetchOnWindowFocus={true} to the component tried to find a similar solution, but it doesn't work
since you're using rtk query, you can update your apiSlice baseQuery function, to check for auth errors and redirect on that, my suggestion is this:
create a base query where you check for the 401 and any other error you want:
// try to execute the req, if it fails logout, and redirect to login.
const baseQueryWithAuth: BaseQueryFn = async (args, api, extraOptions) => {
const result = await baseQuery(args, api, extraOptions);
if (result.error?.status === 403 || result.error?.status === 401) {
// non authorized, then redirect to login page.
// if we have jwt, here u should update the access token
localStorage.removeItem(TOKEN_KEY_IN_LOCAL_STORAGE);
Router.replace('/auth/login');
}
return result;
};
in the snippet above, when I'm referring to token deletion as logout because the token is already invalid in the DB, so I just need to delete it in the front, so no invalidate request is needed.
the mentioned baseQuery can be done like this:
const baseUrl = `${process.env.NEXT_PUBLIC_API_PROTOCOL}://${process.env.NEXT_PUBLIC_API_HOST}/api`;
const TOKEN_KEY_IN_LOCAL_STORAGE = 'SavedToken';
const baseQuery = fetchBaseQuery({
baseUrl,
// credentials: 'include',
prepareHeaders: (headers) => {
// get the authentication token from local storage if it exists
const token = localStorage.getItem(TOKEN_KEY_IN_LOCAL_STORAGE);
if (token) {
headers.set('Authorization', token);
} else {
Router.replace('/auth/login');
}
return headers;
},
});
and then now since you have a working base query with auth support, you can use that to create a main rtk query apiSlice for your project:
// create api
export const apiSlice = createApi({
baseQuery: baseQueryWithAuth,
tagTypes: ['tag1', 'tag2', 'tag3'],
endpoints: (_builder) => ({}),
});
I'm trying to use puppeteer in order to fill the cart in the online store.
(is that legal, right?)
Anyway, the action should post the payload to the store server and I'm doing so in this way.
This code run after login to the system as user.
await page.setRequestInterception(true);
page.on("request", async (request) => {
if (request.url() === "https://www.store.com/server/carts/?userID=xyz&appId=x&loyalty=z") {
const res = await request.continue({
method: "POST",
postData: JSON.stringify(cartContent), // cartcontent is arr of object
headers: { ...request.headers(), ...moreHeaders }, // see note bellow the code
});
console.log(request.headers());
} else {
request.abort();
}
});
await page.goto("https://www.store.com/server/carts/?userID=xyz&appId=x&loyalty=z");
await browser.close();
I don't know why, but nothing actually happened. I believe is up to the request headers.
so I added the request.headers(), but when I log those headers they missing some other headers and contnent not same. so I hard-coded them from the browser when I'm logged in as a user. but still i fill i have lake of knowlege that i will glad to learn.
Headers from puppter (request.headers())
upgrade-insecure-request,user-agent,sec-ch-ua,sec-ch-ua-mobile,sec-ch-ua-platform,accept,cookie
Headers from browswer contains :
:authority,:method,path,scheme,accept,accept-encoding,accept-language ,authorization, content-length,content-type,cookie,origin,referer,
sec-ch-ua,sec-ch-ua,sec-ch-ua-platform,sec-fetch-dest,sec-fetch-mode,
sec-fetch-site,sec-fetch-site,user-agent,x-http-method-override
My backend service is deployed on cloud run.
It can only access by authorized request with IAM service account.
So i use google-auth-libray to request from frontend.
const client = await auth.getIdTokenClient(process.env.NEXT_PUBLIC_CALL_URL!);
const data = await client.request({
url: process.env.NEXT_PUBLIC_CALL_URL!,
});
this is my source.
But I want call with my login token.
So i attempt like this.
const client = await auth.getIdTokenClient(process.env.NEXT_PUBLIC_CALL_URL!);
const header = await client.getRequestHeaders();
const data = await axios.get(process.env.NEXT_PUBLIC_CALL_URL, {
header: {
Authorization: `${header}, Bearer mytoken`
}
})
But It's not working.
Can i call request from front-end like this?
I'm making a automation script for filling a form online with puppeteer, and to not blacklist ip's I decided to use proxies for each request, this is the part which gives me error
console.log(`profile ${ii} started`)
let proxy = await proxy_rotation(proxy_data, ii);
console.log("using proxy: ", proxy);
let exec_path = path.resolve(path.dirname(process.execPath) + "/node_modules/puppeteer/.local-chromium/win64-869685/chrome-win/chrome.exe")
const browser = await puppeteer.launch({
executablePath: exec_path,
args: ['--disable-web-security']
});
const page = await browser.newPage();
console.log("1");
await page.setRequestInterception(true);
await useProxy(page, proxy);
console.log("2");
await page.goto(data[ii][0]); //this is where the error gets thrown
this part below doesn't get to run when using a proxy, without it, runs smotthly
console.log("3");
await page.type("#name", data[ii][1]);
await page.type("#yourEmail", data[ii][2]);
await page.type("#phone", data[ii][3]);
await page.type("#street", data[ii][4]);
await page.type("#city", data[ii][5]);
await page.type("#psc", data[ii][6]);
await page.select('select#state', data[ii][7]);
await page.select('select#prefered_size_sel', data[ii][8]);
await page.$eval('input[name="agreed_personal_info_tiny_contact_form"]', check => check.checked = true);
await page.evaluate(() => {
document.querySelector('input[name="agreed_personal_info_tiny_contact_form"]').click();
});
I just console logged a few numbers, to debug where the script is getting stuck. I also tested the proxy and website I'm trying to access both with a proxy tester and manually, and had no problem accessing it
but when I run my script I get this
I understand it pretty much says it cannot access the url, but there should be no reason for that. Do I need to change the way I'm acessing the url when using a proxy? Or add some extra args to the browser? Can I get a more specific error message somehow? Thanks for any suggestions
Also this is the puppeteer function that throws the error
async function navigate(client, url, referrer, frameId) {
try {
const response = await client.send('Page.navigate', {
url,
referrer,
frameId,
});
ensureNewDocumentNavigation = !!response.loaderId;
return response.errorText
? new Error(`${response.errorText} at ${url}`)
: null;
}
catch (error) {
return error;
}
}
That error indicates that something is off how you are using your proxy. Is useProxy your own function or the one from puppeteer-page-proxy? You mention setting the proxy per-request but seems you are setting it for the whole page, is that intentional? The way your proxy is formatted also seems off- check how I do it below.
You can try launching the browser with your proxy server and using page.authenticate() to handle auth. Like this:
let proxy = await proxy_rotation(proxy_data, ii);
const [host, port, username, password] = proxy.split(':');
const parsedProxy = new URL(`http://${username}:${password}#${host}:${port}`);
const browser = await puppeteer.launch({
executablePath: exec_path,
args: ['--disable-web-security', '--ignore-certificate-errors', `--proxy-server=${parsedProxy.host}`]
});
const page = await browser.newPage();
await page.authenticate({
username: parsedProxy.username,
password: parsedProxy.password,
});
Before doing that I would change what you pass to useProxy such that it looks like http://username:pw#host:port (Lines 2-3).
I have a route in express app like this:
router.get('/search_documents_for_home_page', async (req, res) => {
var responses = [];
await Article.find({}).select('image title body').limit(4).sort({ _id:-1 }).then(articles=>{
responses.push([articles]);
});
await Image.find({}).limit(4).sort({ _id:-1 }).then(images=>{
responses.push([images]);
});
await Video.find({}).limit(4).sort({ _id:-1 }).then(videos=>{
responses.push([videos]);
});
await Project.find({}).limit(4).sort({ _id:-1 }).then(projects=>{
responses.push([projects]);
});
res.json(responses);
});
And when the user goes to the home page, a fetch request is sended:
await fetch('/api/search_documents_for_home_page').then(result=>{
return result.json();
}).then(articles=>{
// show the users all of the documents
});
But I want that only my server can fetch this url.
How do I do that?
Im also using pugjs
You can secure your api by requiring some type of authentication
You can add a check to make sure request is coming from your front end, depending on server this can be handled differently (i.e. window.location.origin)
Enable CORS, only prevents browser>browser calls