API Authentication from Google Chrome in Python - python-3.x

I apologize in advance as I'm pretty new to this stuff. I've seen similar questions, but just can't figure out my particular situation.
I'm trying to use an API through Python, but can't figure out how to authenticate and there is no documentation. I use a service that has a website. The website seems to be powered by an API. Therefore, when I trace all of my network traffic through ctrl-shift-I, I can see all the API calls I need to use as I click through the website. So, even though the API isn't documented, I know all of the end points I need.
Once I login to my account via the website, the API is authenticated. I can then make requests in the browser to the API. However, I can't seem to figure out how to authenticate via Python in the requests library. I'm open to any manner of authenticating, and have even tried using cookies from my browser per other suggestions on Stack Overflow, but I'm very unfamiliar with that method.
Am I completely missing something here? Most of these methods of authenticating I've found as solutions in Stack Overflow that seem to work for others.
s = requests.Session()
payload = {'usernameOrEmail': 'XXXXXXX', 'password':'XXXXXXX'}
s.post('https://XXXXXXX', json=payload, headers={'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36'})
s = requests.Session()
payload = {'usernameOrEmail': 'XXXXXXX', 'password':'XXXXXXX'}
s.post('https://XXXXXXX', json=payload)
url = 'https://XXXXXXX'
values = {'usernameOrEmail': 'XXXXXXX',
'password': 'XXXXXXX'}
r = requests.post(url, data=values, verify=False)
I've tried using the URL of the login page, the member overview page that you are forwarded to after logging in, the API URL's/endpoints, etc. I'm 98% sure I have the name value pairs correct for username and password.
I've even tried a simple get request and appending the usernameOrEmail and password fields on the URL.
I've gotten 401's, 404's, and a 405 most recently depending on the URL method combination.
Apologies in advance as I'm sure this is something extremely basic. Am I on the right track with submitting Username and Password or do I need to go the route of using browser cookies?
I'm using Spyder through Anaconda.
Thanks
Updated for Code Producing a 404:
import requests
s = requests.Session()
payload = {'username_or_email': 'XXXXXXX', 'password':'XXXXXXX'}
test=s.post('https://members.onepeloton.com/login', json=payload)
print(test)

There's nothing HTTP related you can't do in Python that you can do in your browser, except for running Javascript.
You're probably missing a request or some parameter. Maybe there has been a cookie set from before you started you browser session.
Try recording and emulating all the requests from an incognito tab, that way you're sure you start with the same state as your Python session.
Maybe we can help you more if you can tell us which website you're trying to authenticate to.

Related

Instagram server side authentication 2019

I just can't get it right.
There are docs for instagram api that says do one call, it will redirect user to a page where he authenticate, then you get code from Redirect Url and now using this code you can get token. That is working and it is just fine. But, what if i want to do a website for myself as personal page and show my own instagram photos filtered by tag to ANYONE. I don't need OAuth, any login. I'm ok with creating Node server and make a call smth like this:
http://api.instagram.com/auth?client_id_or_login=vvinog&password=secretpassword
and get token from this call. But it looks like in the internet no one do this, or i dump?
Can you please advice some tutorial, approach that allows me to do that?
Any code snippets are super welcome.

How to get the real HQ image from the Instagram API?

Firstly, there are already questions to this topic, but none cover up my problem, entirely, because it's either not the data I need or it's not working properly.
There are services like InstaDP that are able to show you the HQ version of any profile picture from Instagram. Now, I wonder how this is possible?
I did some research and were able to find a higher quality URL when accessing https://www.instagram.com/instagramforbusiness/?__a=1 (see profile_pic_url_hd, answered here). However, InstaDP seems to have a backend that returns a different url that redirects to a way higher quality image: https://instadp-cors-222621.appspot.com/get-hd?id=1107766105 (see at hd_profile_pic_url_info, I extracted the ID for the URL from the result of the ?__a=1 link). I tested this with my personal profile and was able to get the image of myself in an outstanding quality of 1024x1024. However, the ?__a=1 link seem only to return a link for my profile picture in 320x320.
Since InstaDP seem to not be the only player who is able to fetch HQ profile pictures I went ahead and compared the backends of those players. It seems that each service seem to have a different URL to the HQ profile picture of the same Instagram account. So my conclusion is that the Instagram API is involved in all that.
So I created a client key at https://www.instagram.com/developer/. I was also able to get my auth token and determine my logged in csrftoken for the X-CSRFToken header. Now my question is how to continue?
I found a few answers to this topic stating I should request https://i.instagram.com/api/v1/users/1107766105/info/, but it always returns the login page as HTML.
I tried a REST client that uses my Chrome cookies and logged into Instagram before, I tried to set my HTTP headers to X-CSRFToken:<mycookietoken> and Content-Type:application/json. (If I don't set the CSRFToken it errors, so I need to add it, but if the header is set I get the HTML again, even when the CSRFToken is correct. I don't get an error when the CSRFToken is wrong.)
I also tried setting the Origin, Referer and Host to trick Instagram in believing the request came from its own window location, without luck. Setting the Host will even cause a 400 bad request. Even adding my access token in the URL had no effect (?access-token=########).
To sum my question up, how do those services obtain the profile pictures in a such a great quality of up to 1024x1024 from the cdninstagram servers?
Although I am late, but this might be of someone's help:
Step 1:
First thing you need to get HD Instagram profile picture is their profile ID. This can be found in the source code of user profile link. For example if you view source of the following link https://www.instagram.com/abdulhaq0/ and search for "logging_page_id" you will get "profilePage_1285389476". The numbers following the profilePage are the ID for this account.
Step 2:
Next you need put ID in the following URL https://i.instagram.com/api/v1/users/{ProfileID}/info/ and open it in browser. In our case link would be https://i.instagram.com/api/v1/users/1285389476/info/
Step 3:
Now on the link above search for "hd_profile_pic_url_info". There you can get the URL of HD Instagram profile picture.
Hope this helped.
Steps:
1. Get the instagram post link. Eg : https://www.instagram.com/p/Bo-Jru-g7Wa/
or if you don't have the link, the instagram api provides you with a permalink option in the result array which for the above link is Bo-Jru-g7Wa
Now just follow add media?size=l after the url ie.,
Result: High quality image url:
https://www.instagram.com/p/Bo-Jru-g7Wa/media?size=l
you can see it in action here: https://jsfiddle.net/nmj1z7wo/fiddle URL
This link can be considered as a shorthand code to instagram image URL's which are very much bigger
I believe those sites like https://instadp.site/ that show the hi-res of the user profile image do not use the official Instagram API.
In the past it was possible to hack Instagram's CDN URLs to change parameters and get the high resolution from them, but nowadays the URLs are signed and if you change any parameters the URL will fail.
So, the only solution they may be using is to emulate a client. There is a popular PHP client for this: https://github.com/mgp25/Instagram-API
The instagram API have updated. Now we will get only the 320x320 sized image from
https://www.instagram.com/{username}/?__a=1
Even if you get the user id from this endpoint the "user info" endpoint does not return the hd url in its response and should pass a header too now.
import requests
def get_user_by_user_id(user_id):
if user_id:
base_url = "https://i.instagram.com/api/v1/users/{}/info/"
headers = {
'user-agent':'Mozilla/5.0 (iPhone; CPU iPhone OS 12_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 Instagram 105.0.0.11.118 (iPhone11,8; iOS 12_3_1; en_US; en-US; scale=2.00; 828x1792; 165586599)'
}
try:
res = requests.get(base_url.format(user_id),headers=headers)
print(res.json())
except Exception as e:
print("getting user failed, due to '{}'".format(e.message))
get_user_by_user_id(userid)
You can check this code and try. Websites like instadp I guess they does not use instagram's official API
The easiest and most reliable way of getting HD (1080x1080) profile pic is Instaloader CLI.
instaloader USERNAME_OF_INTEREST --no-posts --login YOUR_USERNAME

Set redirect URL for Spotify API

I know there have been multiple questions regarding this issue... however, I'm not sure how to handle my case.
I am using spotipy to access the Spotify API. In my python notebook, I entered:
util.prompt_for_user_token('<user_id>',client_id='<client_id>',client_secret='<client_secret>',redirect_uri='localhost:3000/callback/')
On the spotify developer website, I have listed localhost:3000/callback/ as my redirect URL.
When I run the prompt, I am redirected to the spotify page where I would click 'okay' to authorize the account. However, each time I click the 'okay' button, nothing happens. Tried using a separate browser, tried restarting my computer... I'm not sure what to do.
Thank you!
After being redirected, the library should prompt you to copy the URL you're redirected to and paste it back in your python notebook. It then grabs the access token from the URL and uses it to authenticate.

How can I create an authenticated scraper for Amazon product detail in Node.js?

I'm creating a script that grabs all the shipped items from Amazon and notifies me.
Authentication is needed to see the products though.
I've already tried sending a post request through "request" which returns an error because of the cookies and extra parameters needed.
It would be easy using cheerio afterwards to get the data if the authentication works.
Does anyone have any idea on how we can authenticate successfully?
The link from the email is: https://www.amazon.com/ap/signin/185-3199906-8918341?_encoding=UTF8&accountStatusPolicy=P1&openid.assoc_handle=usflex&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.ns.pape=http%3A%2F%2Fspecs.openid.net%2Fextensions%2Fpape%2F1.0&openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.com%2Fgp%2Fyour-account%2Forder-details%2F185-3199906-8918341%3Fie%3DUTF8%26eoid%3D1%253A1%253Arv%252FYwjiYmnOZY9MYltVnDyf2l6p5pMkMx9deoUeiiw%252FKpPrtZrWqs5l1GGQPVb%2520qaJqHXyCkPEpLZnmDZamKkVDWhtu3dKlW5Gx7Uvxtzs0xlPJ25vduijJrPpHt79P%2520RRZHopOtAyOP4s82VLoeeiDQgq%2520FCP540H%2520UYAV7goZQxB29WObWAVh8VveTwEeWenY3sTx8ZI9%252FBLM2BSqS3IUIURW8mzMnAB9t7wglUiAcoR%252FcUhSIx%25201eNV4MspVAp7fLkeANag72BxgmsjFfRhnsxfji1VhZXLawqFeK9SBnvbUfkNWUC%2520IXWh6VcuoStBG3x%2520ZUkzGHw1ORi4J%2520Hg%253D%253D%26orderID%3D105-6914722-5422613%26ref_%3DTE_simp_on_T1&pageId=webcs-yourorder&showRmrMe=1
You cannot guarantee any of the form input values of the sign in page. So you must also scrape the login form.
Here is the process:
In your server, make Request to the URL in your question
Using Cheerio parse the DOM and grab all of the form fields from "#ap_signin_form".
Add in your data (Username/Pass) then make a POST request to the form action "https://www.amazon.com/ap/signin" (This should also be scraped)
Hopefully that will get you past the login screen. You will need to ensure all future requests pass the cookies set from login.
Now this kind of thing is clearly against most TOS's so I would urge caution in doing this kind of thing often.

Is it advisable to configure a URL with both post and get?

I have a link on my page that redirects to another page but the request is sent through POST method. Now when the user refreshes the new page the request is sent through GET method. The URL is just used to display a page. My question is, is it advisable to use both POST and GET for the same URL call or will it cause problems related to security or any other? If so please do explain how.
No. Use POST if the link is executing an unsafe action (e.g. logout, change password) or it is sending sensitive data you do not want displayed in the URL (URLs are logged by default in the browser history and by many appliances, proxies and web servers). POST can also be used when a large amount of information is to be sent.
Otherwise use GET.

Resources