Web Scraping Amazon "See More Buying Options" Node Cheerio Puppeteer - node.js

I am trying to web scrape amazon for GPU prices (How original!). I am an elementary school teacher trying a fun beginner node project to get prices on amazon.
There are three options I have found when scraping... Available Add to cart, See other buying options, and currently unavailable.
I have been able to see if the "add to cart" button is present by checking the length and can scrape the price to see if it's one I'd be willing to pay, then alert me. However, when checking other buying options, I can get to the side pane but I can't scrape the prices of the other buying options because cheerio cannot see a specific ID. Each "buying option" div has an ID="#aod-offer", but even when seeing if it is present, it returns 0, even though I clearly see it in the inspector. I have been practicing on pages that have at least 1 buying option. Not sure why it is not scraping any data.
I have been practicing on this card:
https://www.amazon.com/Gaming-GeForce-Graphics-DisplayPort-Bearings/dp/B08TFLDLTM/ref=sr_1_17?crid=1FL7W42YBL5XB&dchild=1&keywords=geforce+rtx+3080+graphics+card&qid=1617721759&s=electronics&sprefix=GeForce+rtx+3080+graphics+card%2Celectronics%2C173&sr=1-17
//Scrapes amazon pages
async function CheckAvailability(page){
//for(var i = 0; i < items.length; i++){}
await page.goto(items[1].url);
const html = await page.content();
const $ = cheerio.load(html);
//If add to cart button is present
if($("#add-to-cart-button").length > 0){
//Check price of item
const priceStr = $("#price_inside_buybox").text().replace("$", "");
const price = Number(priceStr);
console.log(priceStr);
console.log(price);
//Check price against what you are willing to spend
if(checkPrice(price)){
console.log("You are willing to buy this item");
}else{
console.log("The price is too rich for your blood!");
}
// Check if there are other sellers
}else if ($("#unqualifiedBuyBox").length > 0){
console.log("There might be others");
await page.$eval("a[title ='See All Buying Options']", elem => elem.click());
await page.waitFor(2000);
const offers = [];
const offerListings = $("#aod-offer").each((i, e) => {
offers[i] = $(e).html();
});
console.log(offers);
// Currently unavailable is displayed
}else if ($("#outOfStock").length > 0){
console.log("There are none available, moving on...");
}
}
//Checks a price against what I'm willing to spend. Lowered the number for testing. I know I cannot
get a gpu for $200
function checkPrice(price){
if (price < 200){
return true;
}else{
return false;
}
}

I know my code is probably pretty basic, but I changed my const $ to Let $ and reloaded cheerio with new HTML data in the else if section, and it worked. I thought because I was on the same page that my cheerio did not have to be updated but apparently it did.

Related

Get friend suggestions from users contacts based on whether the contact number is associated with any given account

I am trying to build an expo app with firebase auth and an express backend.
I want to build a friend suggestion system from contacts like in snapchat and instagram and other social media.
I am requesting the users for contacts permission and then processing that array just to get the phone numbers and match the format needed by firebase then I match the numbers with
However, if the contact list is too large then the request takes very long to complete (about 30s).
I understand that all contacts must be checked but how do big social medias reduce the time taken to check contacts by so much.
Server code:
//firebase admin function
const phoneNumberExists = async (phoneNumber: string) =>
await getAuth(admin).getUserByPhoneNumber(phoneNumber);
const getUsersByContacts = async (
//array of contact numbers as a string because some might include the country code
contacts: string[],
//current user's uid
currUid: string
) => {
const users = [];
for await (let contact of contacts) {
try {
//formats string according to indian phone numbers and adds country code
contact = contact.trim().replace(/ /g, "");
if (contact.length !== 13 || !contact.startsWith("+")) {
if (contact.length === 10) contact = "+91" + contact;
else if (contact.length === 12) contact = "+" + contact;
}
const { uid } = await phoneNumberExists(contact);
//checks whether mobile number belongs to current user
if (currUid !== uid) {
// gets properties of the user based on uid
const actualUser = await getUserData(uid);
if (actualUser) users.push(actualUser);
}
} catch (error) {
// this error code means no user found with the phone number so I continue
if (error.code === "auth/user-not-found") continue;
}
}
return users;
};
Every contact has to be checked and since it's the firebase function which is causing the delay I was wondering if there was a way to this with node js duplex stream. I haven't worked with streams in the past but from what I read about streams they fit the need. However all examples of streams are related to file streaming.
Is it possible to send contacts as a stream and get suggestions as a stream.
Should I convert the contacts array into a file first and then stream it?

How to handle multiple requests to purchase at same time

Let's imagine we have a shopping website which the users are able to purchase items with their account-balance.
user A requests to purchase item B.
here are the steps:
Database gets user's balance.
Checks if balance is more than the cost.
Updates user's balance ( balance - cost = newBalance )
now this is where my question begins:
since javascript is single-threaded and we are requesting to db in an asynchronous way what happens if:
user A sends multiple requests to purchase the item
another user tries to purchase the item ( item should be out of stock after 1 purchase )
I've done some testing and I want to know what is the best way to prevent userA to purchase multiple items when in reality he should be out of balance after second purchase?
my test:
const fs = require('fs');
const cost = 500;
// .data.txt has `1000` as its content
function getMoneyFromDatabase() {
return new Promise((resolve, reject) => {
fs.readFile('./data.txt', 'utf-8', (err, data) => {
if (err) reject(err);
else resolve(Number(data));
});
});
}
function setMoneyToDatabase(newMoney) {
return new Promise((resolve, reject) => {
fs.writeFile('./data.txt', newMoney, (err) => {
if (err) reject(err);
else resolve();
});
});
}
async function getMoney() {
const money = await getMoneyFromDatabase();
if (money >= cost) {
// Able to purchase
console.log('able to purchase');
const newMoney = money - cost;
await setMoneyToDatabase(newMoney);
console.log('purchased');
}
}
getMoney();
getMoney();
getMoney();
getMoney();
this logs 4 purchased while in reality it should only be able to purchase two times
and the data saved inside data.txt is 500 while it should be -1000 ( 500, 0, -500, -1000 )
One thing you can do is make sure your API requests related to charging are idempotent, meaning they can be sent multiple times but will result in the same outcome as long as the request is the same.
You can implement idempotent requests by sending a unique key along with the request to make sure that only the first time it will be accepted as a new request otherwise it will be handled as a duplicate request. This is also useful when you want to safely retry a requests without accidentally charging the user multiple times.
Many payment processors have built-in support for this such as Stripe: https://stripe.com/docs/api/idempotent_requests
As for stock, you should only decrease it when a payment is verified or if it's not automatic the first user with a pending invoice for it. Otherwise the stock will go down even if the user for example has the balance but payment could not be processed.
Implementing payments is hard and can only be learned to be implemented in a decent way after trial and errors, I'd recommend you try to do a minimal app and handle payments with a payment gateway such as Stripe and advance by handling edge cases.

dialogflow Webhookclient "request_" property

I am trying to build up a facebook messenger chatbot using Dialogflow. In the dialogflow fulfillment inline editor, I found that I can use agent.request_.body to get the body of the request. I assume "request_" is a property of WebhoodClient object? But I couldn't find any documentation elaborate that, could you please advise if my understanding is correct and where I can find the reference or documentation?
const agent = new WebhookClient({ request, response });
console.log(JSON.stringify(agent.request_.body));
Thanks
Google provides documentation for Dialogflow webhooks here, which include this sample webhook to inspect parameters and dynamically create slot filling prompts:
exports.dialogflowFirebaseFulfillment = functions.https.onRequest((request, response) => {
const agent = new WebhookClient({ request, response });
function flight(agent) {
const city = agent.parameters['geo-city'];
const time = agent.parameters['time'];
const gotCity = city.length > 0;
const gotTime = time.length > 0;
if(gotCity && gotTime) {
agent.add(`Nice, you want to fly to ${city} at ${time}.`);
} else if (gotCity && !gotTime) {
agent.add('Let me know which time you want to fly');
} else if (gotTime && !gotCity) {
agent.add('Let me know which city you want to fly to');
} else {
agent.add('Let me know which city and time you want to fly');
}
}
let intentMap = new Map();
intentMap.set('flight', flight);
agent.handleRequest(intentMap);
});
My guess would be to add
console.log(agent);
right before defining the flight function, then checking the logs to see which objects agent contains, then adding iterations of console.log(agent.fakeObjectName) until you find the information you're looking for.
If you're following the deployment process recommended in Actions on Google's Codelabs level 2, your logs will show up in the Firebase console, like this:
Hope that helps!
Just a note.
I had a code similar to this:
const city = agent.parameters['geo-city'];
There is an icon that suggest it's better written in dot notation.
that is gone after I changed it to:
const city = agent.parameters.geo-city;

Can we Post content on Instagram on user's behalf? [duplicate]

I am building a php application which needs to post the user uploaded picture directly to Instagram, but after a quick search i found that there is no such function in the API :( and it feels weird... because they should provide one. I am not sure if there is any other way (except the apps for android and iOS) to upload picture using php. Kindly give me any sort of idea if there is any possibility.
I also read this ,
How do I share a link and photo with Instagram using PHP
Update:
Instagram are now banning accounts and removing the images based on this method. Please use with caution.
It seems that everyone who has answered this question with something along the lines of it can't be done is somewhat correct. Officially, you cannot post a photo to Instagram with their API. However, if you reverse engineer the API, you can.
function SendRequest($url, $post, $post_data, $user_agent, $cookies) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://i.instagram.com/api/v1/'.$url);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
if($post) {
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
}
if($cookies) {
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');
} else {
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
}
$response = curl_exec($ch);
$http = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
return array($http, $response);
}
function GenerateGuid() {
return sprintf('%04x%04x-%04x-%04x-%04x-%04x%04x%04x',
mt_rand(0, 65535),
mt_rand(0, 65535),
mt_rand(0, 65535),
mt_rand(16384, 20479),
mt_rand(32768, 49151),
mt_rand(0, 65535),
mt_rand(0, 65535),
mt_rand(0, 65535));
}
function GenerateUserAgent() {
$resolutions = array('720x1280', '320x480', '480x800', '1024x768', '1280x720', '768x1024', '480x320');
$versions = array('GT-N7000', 'SM-N9000', 'GT-I9220', 'GT-I9100');
$dpis = array('120', '160', '320', '240');
$ver = $versions[array_rand($versions)];
$dpi = $dpis[array_rand($dpis)];
$res = $resolutions[array_rand($resolutions)];
return 'Instagram 4.'.mt_rand(1,2).'.'.mt_rand(0,2).' Android ('.mt_rand(10,11).'/'.mt_rand(1,3).'.'.mt_rand(3,5).'.'.mt_rand(0,5).'; '.$dpi.'; '.$res.'; samsung; '.$ver.'; '.$ver.'; smdkc210; en_US)';
}
function GenerateSignature($data) {
return hash_hmac('sha256', $data, 'b4a23f5e39b5929e0666ac5de94c89d1618a2916');
}
function GetPostData($filename) {
if(!$filename) {
echo "The image doesn't exist ".$filename;
} else {
$post_data = array('device_timestamp' => time(),
'photo' => '#'.$filename);
return $post_data;
}
}
// Set the username and password of the account that you wish to post a photo to
$username = 'ig_username';
$password = 'ig_password';
// Set the path to the file that you wish to post.
// This must be jpeg format and it must be a perfect square
$filename = 'pictures/test.jpg';
// Set the caption for the photo
$caption = "Test caption";
// Define the user agent
$agent = GenerateUserAgent();
// Define the GuID
$guid = GenerateGuid();
// Set the devide ID
$device_id = "android-".$guid;
/* LOG IN */
// You must be logged in to the account that you wish to post a photo too
// Set all of the parameters in the string, and then sign it with their API key using SHA-256
$data ='{"device_id":"'.$device_id.'","guid":"'.$guid.'","username":"'.$username.'","password":"'.$password.'","Content-Type":"application/x-www-form-urlencoded; charset=UTF-8"}';
$sig = GenerateSignature($data);
$data = 'signed_body='.$sig.'.'.urlencode($data).'&ig_sig_key_version=4';
$login = SendRequest('accounts/login/', true, $data, $agent, false);
if(strpos($login[1], "Sorry, an error occurred while processing this request.")) {
echo "Request failed, there's a chance that this proxy/ip is blocked";
} else {
if(empty($login[1])) {
echo "Empty response received from the server while trying to login";
} else {
// Decode the array that is returned
$obj = #json_decode($login[1], true);
if(empty($obj)) {
echo "Could not decode the response: ".$body;
} else {
// Post the picture
$data = GetPostData($filename);
$post = SendRequest('media/upload/', true, $data, $agent, true);
if(empty($post[1])) {
echo "Empty response received from the server while trying to post the image";
} else {
// Decode the response
$obj = #json_decode($post[1], true);
if(empty($obj)) {
echo "Could not decode the response";
} else {
$status = $obj['status'];
if($status == 'ok') {
// Remove and line breaks from the caption
$caption = preg_replace("/\r|\n/", "", $caption);
$media_id = $obj['media_id'];
$device_id = "android-".$guid;
$data = '{"device_id":"'.$device_id.'","guid":"'.$guid.'","media_id":"'.$media_id.'","caption":"'.trim($caption).'","device_timestamp":"'.time().'","source_type":"5","filter_type":"0","extra":"{}","Content-Type":"application/x-www-form-urlencoded; charset=UTF-8"}';
$sig = GenerateSignature($data);
$new_data = 'signed_body='.$sig.'.'.urlencode($data).'&ig_sig_key_version=4';
// Now, configure the photo
$conf = SendRequest('media/configure/', true, $new_data, $agent, true);
if(empty($conf[1])) {
echo "Empty response received from the server while trying to configure the image";
} else {
if(strpos($conf[1], "login_required")) {
echo "You are not logged in. There's a chance that the account is banned";
} else {
$obj = #json_decode($conf[1], true);
$status = $obj['status'];
if($status != 'fail') {
echo "Success";
} else {
echo 'Fail';
}
}
}
} else {
echo "Status isn't okay";
}
}
}
}
}
}
Just copy and paste the code above in your text editor, change the few variables accordingly and VOILA! I wrote an article about this and I've done it many times. See a demo here.
If you read the link you shared, the accepted answer is:
You cannot post pictures to Instagram via the API.
Instagram has now said:
Now you can post your content using Instagram's APIs (New) effects from
26th Jan 2021!
https://developers.facebook.com/blog/post/2021/01/26/introducing-instagram-content-publishing-api/
Instagram now allows businesses to schedule their posts, using the new Content Publishing Beta endpoints.
https://developers.facebook.com/blog/post/2018/01/30/instagram-graph-api-updates/
However, this blog post - https://business.instagram.com/blog/instagram-api-features-updates - makes it clear that they are only opening that API to their Facebook Marketing Partners or Instagram Partners.
To get started with scheduling posts, please work with one of our
Facebook Marketing Partners or Instagram Partners.
This link from Facebook - https://developers.facebook.com/docs/instagram-api/content-publishing - lists it as a closed beta.
The Content Publishing API is in closed beta with Facebook Marketing
Partners and Instagram Partners only. We are not accepting new
applicants at this time.
But this is how you would do it:
You have a photo at...
https://www.example.com/images/bronz-fonz.jpg
You want to publish it with the hashtag "#BronzFonz".
You could use the /user/media edge to create the container like this:
POST graph.facebook.com
/17841400008460056/media?
image_url=https%3A%2F%2Fwww.example.com%2Fimages%2Fbronz-fonz.jpg&
caption=%23BronzFonz
This would return a container ID (let's say 17889455560051444), which you would then publish using the /user/media_publish edge, like this:
POST graph.facebook.com
/17841405822304914/media_publish
?creation_id=17889455560051444
This example from the docs.
I tried using IFTTT and many other services but all were doing things or post from Instagram to another platform not to Instagram. I read more to found Instagram does not provide any such API as of now.
Using blue stack is again involving heavy installation and doing things manually only.
However, you can use your Google Chrome on the desktop version to make a post on Instagram. It needs a bit tweak.
Open your chrome and browse Instagram.com
Go to inspect element by right clicking on chrome.
From top right corener menu drop down on developer tools, select more tool.
Further select network conditions.
In the network selection section, see the second section there named user agent.
Uncheck select automatically, and select chrome for Android from the list of given user agent.
Refresh your Instagram.com page.
You will notice a change in UI and the option to make a post on Instagram.
Your life is now easy.
Let me know an easier way if you can find any.
I wrote on https://www.inteligentcomp.com/2018/11/how-to-upload-to-instagram-from-pc-mac.html about it.
Working Screenshot
For anyone who is searching for a solution about posting to Instagram using AWS lambda and puppeteer (chrome-aws-lambda). Noted that this solution allow you to post 1 photo for each post only. If you are not using lambda, just replace chrome-aws-lambda with puppeteer.
For the first launch of lambda, it is normal that will not work because instagram detects “Suspicious login attempt”. Just goto instagram page using your PC and approve it, everything should be fine.
Here's my code, feel free to optimize it:
// instagram.js
const chromium = require('chrome-aws-lambda');
const username = process.env.IG_USERNAME;
const password = process.env.IG_PASSWORD;
module.exports.post = async function(fileToUpload, caption){
const browser = await chromium.puppeteer.launch({
args: [...chromium.args, '--window-size=520,700'],
defaultViewport: chromium.defaultViewport,
executablePath: await chromium.executablePath,
headless: false,
ignoreHTTPSErrors: true,
});
const page = await browser.newPage();
await page.setUserAgent('Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X) AppleWebKit/603.2.4 (KHTML, like Gecko) FxiOS/7.5b3349 Mobile/14F89 Safari/603.2.4');
await page.goto('https://www.instagram.com/', {waitUntil: 'networkidle2'});
const [buttonLogIn] = await page.$x("//button[contains(., 'Log In')]");
if (buttonLogIn) {
await buttonLogIn.click();
}
await page.waitFor('input[name="username"]');
await page.type('input[name="username"]', username)
await page.type('input[name="password"]', password)
await page.click('form button[type="submit"]');
await page.waitFor(3000);
const [buttonSaveInfo] = await page.$x("//button[contains(., 'Not Now')]");
if (buttonSaveInfo) {
await buttonSaveInfo.click();
}
await page.waitFor(3000);
const [buttonNotificationNotNow] = await page.$x("//button[contains(., 'Not Now')]");
const [buttonNotificationCancel] = await page.$x("//button[contains(., 'Cancel')]");
if (buttonNotificationNotNow) {
await buttonNotificationNotNow.click();
} else if (buttonNotificationCancel) {
await buttonNotificationCancel.click();
}
await page.waitFor('form[enctype="multipart/form-data"]');
const inputUploadHandle = await page.$('form[enctype="multipart/form-data"] input[type=file]');
await page.waitFor(5000);
const [buttonPopUpNotNow] = await page.$x("//button[contains(., 'Not Now')]");
const [buttonPopUpCancel] = await page.$x("//button[contains(., 'Cancel')]");
if (buttonPopUpNotNow) {
await buttonPopUpNotNow.click();
} else if (buttonPopUpCancel) {
await buttonPopUpCancel.click();
}
await page.click('[data-testid="new-post-button"]')
await inputUploadHandle.uploadFile(fileToUpload);
await page.waitFor(3000);
const [buttonNext] = await page.$x("//button[contains(., 'Next')]");
await buttonNext.click();
await page.waitFor(3000);
await page.type('textarea', caption);
const [buttonShare] = await page.$x("//button[contains(., 'Share')]");
await buttonShare.click();
await page.waitFor(3000);
return true;
};
// handler.js
await instagram.post('/tmp/image.png', '#text');
it must be local file path, if it is url, download it to /tmp folder first.
Updated:
Instagram is blocking all suspicious login attempt now unless you approve it manually every time it executed. To solve that, better to save your cookies as json and import it to puppeteer.
For users who find this question, you can pass photos to the instagram sharing flow (from your app to the filters screen) on iPhone using iPhone hooks: http://help.instagram.com/355896521173347 Other than that, there is currently no way in version 1 of the api.
If it has a UI, it has an "API". Let's use the following example: I want to publish the pic I use in any new blog post I create. Let's assume is Wordpress.
Create a service that is constantly monitoring your blog via RSS.
When a new blog post is posted, download the picture.
(Optional) Use a third party API to apply some overlays and whatnot to your pic.
Place the photo in a well-known location on your PC or server.
Configure Chrome (read above) so that you can use the browser as a mobile.
Using Selenium (or any other of those libraries), simulate the entire process of posting on Instagram.
Done. You should have it.
There is no API to post photo to instagram using API , But there is a simple way is that install google extension " User Agent " it will covert your browser to android mobile chrome version . Here is the extension link https://chrome.google.com/webstore/detail/user-agent-switcher/clddifkhlkcojbojppdojfeeikdkgiae?utm_source=chrome-ntp-icon
just click on extension icon and choose chrome for android and open Instagram.com

Chrome inapp purchase test

I'm trying to implement an inapp purchase for my Chrome Extension. I have followed the guide on https://developer.chrome.com/webstore/payments-iap, added 1 buyable item and implemented following code on an onclick event:
google.payments.inapp.getSkuDetails({
'parameters': {'env': 'prod'},
'success': onSkuDetails,
'failure': onSkuDetailsFail
});
var onSkuDetails = function (response) {
console.log("onSkuDetails", response);
/*var products = response.response.details.inAppProducts;
var count = products.length;
for (var i = 0; i < count; i++) {
var product = products[i];
addProductToUI(product);
}
getLicenses();*/
}
var onSkuDetailsFail = function(result){
console.log("onSkuDetailsFailed", result);
}
But every time I click it, I get
onSkuDetailsFailed > "INVALID_RESPONSE_ERROR"
I have added the buy.js and changed some permissions, but still don't see where the problem might be.
Do note, that I'm testing this extension locally and not from the Chrome webstore using my own account and a friend's account which I included as testperson on the dashboard.

Resources