I'm trying to use puppeteer in order to fill the cart in the online store.
(is that legal, right?)
Anyway, the action should post the payload to the store server and I'm doing so in this way.
This code run after login to the system as user.
await page.setRequestInterception(true);
page.on("request", async (request) => {
if (request.url() === "https://www.store.com/server/carts/?userID=xyz&appId=x&loyalty=z") {
const res = await request.continue({
method: "POST",
postData: JSON.stringify(cartContent), // cartcontent is arr of object
headers: { ...request.headers(), ...moreHeaders }, // see note bellow the code
});
console.log(request.headers());
} else {
request.abort();
}
});
await page.goto("https://www.store.com/server/carts/?userID=xyz&appId=x&loyalty=z");
await browser.close();
I don't know why, but nothing actually happened. I believe is up to the request headers.
so I added the request.headers(), but when I log those headers they missing some other headers and contnent not same. so I hard-coded them from the browser when I'm logged in as a user. but still i fill i have lake of knowlege that i will glad to learn.
Headers from puppter (request.headers())
upgrade-insecure-request,user-agent,sec-ch-ua,sec-ch-ua-mobile,sec-ch-ua-platform,accept,cookie
Headers from browswer contains :
:authority,:method,path,scheme,accept,accept-encoding,accept-language ,authorization, content-length,content-type,cookie,origin,referer,
sec-ch-ua,sec-ch-ua,sec-ch-ua-platform,sec-fetch-dest,sec-fetch-mode,
sec-fetch-site,sec-fetch-site,user-agent,x-http-method-override
Related
I am dealing with a request that when called in a browser, will call multiple requests, I managed to capture these requests using puppeteer, but I don't know how to do it in Axios.
for example, make a GET request using Axios to "www.example.com/welcome", this will call about 10 requests inside that request.
in puppeteer, I can do the following to solve this issue:
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', async request => {
const auth = request.headers().authorization;
if (auth) {
TOKEN = auth ? auth.substring(6, auth.length) : TOKEN;
await request.abort();
} else {
await request.continue();
}
});
is there a way to intercept these requests and capture the headers of each of them?
...
const userIcon = document.querySelector('.user-icon');
userIcon.addEventListener("click", () => {
if (token) {
fetch('/privatearea', {
headers: {
'authorization': `Bearer ${token}`
}
}).catch(err => console.log(err));
} else {
window.location.assign('/login');
}
...
What i'm trying to do is:
When the element "userIcon" is clicked if "token" exists, i want to fetch the url "http://localhost:3000/privatearea".
If token doesn't exist the url "http://localhost:3000/login" is reached.
...
router.get('/privatearea', authenticateToken, (req, res) => {
res.render("private-area");
});
...
The backend has been done with node js and express.
So, if i click the element the url "http://localhost:3000/privatearea" should be reached and the page "private-area" should be renderized.
But it doesn't happen. I don't need a response, but i would reach the url by setting some headers.
fetch() by itself does not render anything in the browser. It just returns content back to your Javascript. So, if you had a .then() handler on the fetch() call, you would just get back the rendered data from res.render("private-area"); on your server.
userIcon.addEventListener("click", () => {
if (token) {
fetch('/privatearea', {
headers: {
'authorization': `Bearer ${token}`
}
}).then(data => {
// here you will see the result of res.render("private-area");
// on your server
console.log(data);
}).catch(err => console.log(err));
} else {
window.location.assign('/login');
}
...
}
So, the way you have it isn't the correct design for using fetch(). Since you don't show the larger context of this feature here, it's not entirely clear what the best architecture/solution would be here. Here are some options:
You could take the data from the .then() handler I show above and insert it into the current page using your own Javascript.
You could do window.location = '/privatearea'; and find some other way to communicate the Bearer token to your server (perhaps a previous call to your server sets a session cookie) so your server can use the session cookie to verify that the browser is allowed to enter /privatearea.
You could switch your auth model to something that is more friendly to browser URLs (like a session/cookie) since you can't use a Bearer Token from a URL entered into the browser bar as they can only be used from Javascript. The user logs in, gets assigned a session cookie and URL handlers on your server for URLs from the browser check that.
I have a function in Node that needs to fetch all of the Users from Azure AD (I'm utilizing Microsoft Graph and Axios).
This function queries Microsoft Graph, the response includes an array of users. The max number of objects in this array is capped at 999. If there are more users, this response will also include an #odata.nextLink. My plan was to query, and if there is an #odata.nextLink in the response, query again until there is no #odata.nextLink. Not entirely sure if there is another way to do this better.
async function fetchUsers(tokenEndpoint, client_id, client_secret, query) {
try {
let allUsers = [];
let bearer = `Bearer ${await fetchToken(
tokenEndpoint,
client_id,
client_secret
)}`;
let res = await fetchData(bearer, `${msAPI}${query}`);
if (res.status === 200) {
allUsers.push(...res.data.value);
console.log(allUsers.length);
let nextURL = res.data["#odata.nextLink"];
while (nextURL != undefined) {
bearer = `Bearer ${await fetchToken(
tokenEndpoint,
client_id,
client_secret
)}`;
res = await fetchData(bearer, nextURL);
allUsers.push(...res.data.value);
nextURL = res.data["#odata.nextLink"];
console.log(allUsers.length);
}
}
return allUsers;
} catch (err) {
handleError(err);
}
}
async function fetchToken(tokenEndpoint, client_id, client_secret) {
try {
const res = await axios.post(
tokenEndpoint,
qs.stringify({
client_id,
client_secret,
grant_type: "client_credentials",
scope
})
);
return res.data.access_token;
} catch (err) {
handleError(err);
}
}
async function fetchData(bearer, url) {
try {
const res = await axios.get(url, {
headers: {
Authorization: bearer
}
});
return res;
} catch (err) {
handleError(err);
}
}
I have this statement in another async function
const allUsers = await fetchUsers(tokenEndpoint,clientId,clientSecret,query);
Sometimes this runs perfectly and I get all my Azure AD users and sometimes I receive an ECONNRESET error in the middle of the function being run and it won't finish. This error seemed to happen a few times on one network, then I tested on a different network and didn't get the error once. Swapped back to the original network and got the errors again. Not sure if this is a network issue or the way I am doing the Axios requests in my loop.
Regardless, how would I go about resolving this ECONNRESET issue?
An ECONNRESET occurs when a TCP abruptly closed. This could be caused by a failure in your local network, the remote service, or anywhere in between the two.
The most likely cause here is requesting $top=999. That is an excessively large (and therefore long) page size. You should be setting this to a smaller value or, ideally, leaving it to the default. This ensures the page sizes are reasonable, requests complete quickly, and helps avoid potential getting throttled by AAD.
Your "follow each #odata.nextLink plan is the correct pattern to use. This allows you to pull a page of data and process it prior to making subsequent requests for additional pages. In between requests you'll want to store the current #odata.nextLink so if anything goes wrong, you can pick up where you left off.
I am building an analytics dashboard using the MERN stack (Express, Node) are the Important things to highlight.
As part of a dash view, I was trying to find if it's possible to trigger a PhantomJS call to create a pdf report using a button on the page itself.
Given you need to be logged in to see your own analytics, I can not just run phantom from the command line and pass it in the URL of one of the dashboard pages since it requires a login and queries to be made.
Is it possible to do this with phantomJS?
If I correctly understood your question.
Example:
[main.js]
const dashboardToPdfCtrl = require("./controllers/phantom/pdf");
router.route("/api/dashboard/phantom").post(dashboardToPdfCtrl.createPdf);
router.route("/api/dashboard/phantom/html")
.post(dashboardToPdfCtrl.createDashboard);
When the user clicks on the "button" you can validate the USER according to the architecture of your application.
[pdf.js]
exports.createPdf= async (req, res) => {
if (!req.user || !req.user.sub) {
return res
.status(401)
.send({ message: 'No authorization token was found' });
}
const instance = await phantom.create();
const page = await instance.createPage();
const settings = {
operation: "POST",
encoding: "utf8",
headers: {
"Content-Type": "application/json"
},
data: JSON.stringify({
user: req.body.userId,
anyDataYouNeedToRender: req.body.anyDataYouNeedToRender
})
};
//POST request to /api/dashboard/phantom/html
await page.open(
`${process.env.HOST}:${
process.env.PORT
}/api/dashboard/phantom/html`,
settings
);
//Save the content of /public/dashboard/dashboard.html with received data to pdf
const pageSaved = await page.render(
path.resolve(`./public/dashboard/file.pdf`)
);
if (pageSaved) await instance.exit();
}
exports.createDashboard = (req, res) => {
res.render(
path.resolve("./public/dashboard/dashboard.html"),
{ user: req.body.user,
anyDataYouNeedToRender: req.body:anyDataYouNeedToRender
}
);
};
Is that what you were looking for? I want to help you, feel free to ask detalization.
P.S. As friends told before in comments, it will be great if you give us more information to understend you goal.
My req payload to the Slack API using the icon_emoji feature of this API is not correct. The expected behavior is to have an emoji being :smile: to be displayed as my Slackbot's icon image whenever the bot posts a message to the channel. The current behavior is a default Slack image instead of the :smile:. I don't presently see what I am doing wrong. This is my fourth attempt across months at correcting this possibly so I would appreciate any advice here.
Here is my code, "as_user" has to be set to true for this to work per the documentation for posting messages.
Here is my index.js file:
const fetch = require("node-fetch"),
config = require("../config.js"),
icon = ":smile:";
module.exports = {
postMessage: (message) => {
if(!config.SLACK_CONFIG.webhook_url) {
throw Error("Please set SLACK_MEETUP_WEBHOOK_URL");
}
return fetch(config.SLACK_CONFIG.webhook_url, {
method: "POST",
headers: {
"Content-Type": "application/json"
},
body: JSON.stringify({ text: message, "as-user": false, "icon-emoji": icon })
}).then(res => res.text()).then((text) => {
if(text !== "ok") {
throw Error("Failed to post message to slack");
}
});
}
};
I faced this problem, I needed to add write customize permissions to the slack app (doc)
To add it:
Go to slack apps page and select the app
Go to OAuth & Permissions (under Features section)
In Scope subsection add chat:write.customize permission
I used the slack-node NPM module, and my code looks like
slack = new Slack();
slack.setWebhook('https://hooks.slack.com/services/JUMBLE/JUMBLE/ALONGERJUMBLE');
slack.webhook({
channel: "#mychannel",
username: "nametopostunder",
text: content,
icon_emoji: ":ship:",
}
My ship emoji posts successfully. If you want to roll your own, you could probably go read the slack-node module's code at https://github.com/clonn/slack-node-sdk#readme and figure out what magic they're using.
I didn't check slack-node, this is just an alternative solution with plain nodejs.
I tried a lot of combinations using 'hooks' endpoint, unfortunately it didn't work.
Instead of 'hooks' I used https://slack.com/api/chat.postMessage with slack token.
const https = require('https');
const slackToken = 'xoxb-XXXXX-XXX'
const data = JSON.stringify({
username: 'someUsername',
icon_emoji: ':+1:',
channel: '#your_channel',
token: slackToken,
text: 'helloWorld'
});
const options = {
hostname: 'slack.com',
port: 443,
path: '/api/chat.postMessage',
method: 'POST',
headers: {
'Content-Type': 'application/json',
authorization: `Bearer ${slackToken}`,
},
};
const req = https.request(options, res => {
console.log(`statusCode: ${res.statusCode}`);
res.on('data', d => {
process.stdout.write(d);
});
});
req.on('error', error => {
console.error(error);
});
req.write(data);
req.end();
Did you check your display settings in slack? I encountered the same issue and changed the setting in Preferences => Messages & Media => Theme from Compact to Clean and the icon is displayed as expected.
Sometimes the icon for a message won't change becuase of 2 reasons:
First, the chat:write.customize permission is not granted as defined in the documentation:
https://api.slack.com/methods/chat.postMessage#authorship
In that case you won't get warnings or errors - just nothing happens.
Second - the because the username for the bot remains the same, the icons and messages are collapsed. This is true for both the (android) mobile app and the desktop app. On the mobile app however, you can click a message, see the detailview of the message and see the image that belongs to that particular message.
As we can see in that image, the first round of messages get the :information_source: icon, the icon/usernames of the rest of the messages are collapsed, but with different usernames the icons are visible.