I'm getting a click event from SendGrid for every link in an email sent to Gmail users, and it is definitely not me clicking the links, I even get the first click event to my SendGrid webhook before SendGrid sends the delivered event to me.
This adds needless hits to both the webhook endpoint and to the pages it hits, I can filter out the click events so I won't get false click counts by filtering for user agents with "Gmail-content-sampling" in it
as the user agent supplied is
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246
Gmail-content-sampling
However it would still needlessly hit the site umpteen thousand times in a few minutes (I don't think that will agree with the standard click here to validate email address links) and I don't want to have to add a block user agent per for every request.
Although filtering would be a problem if other providers do the same thing or Gmail change the user agent.
I have tried adding rel="nofollow" to the links but it still does it.
Anyone experience this/know how to get gmail to f-off ?
Edit:
As at 10th August google have undone their stuff up.
Related
I apologize in advance as I'm pretty new to this stuff. I've seen similar questions, but just can't figure out my particular situation.
I'm trying to use an API through Python, but can't figure out how to authenticate and there is no documentation. I use a service that has a website. The website seems to be powered by an API. Therefore, when I trace all of my network traffic through ctrl-shift-I, I can see all the API calls I need to use as I click through the website. So, even though the API isn't documented, I know all of the end points I need.
Once I login to my account via the website, the API is authenticated. I can then make requests in the browser to the API. However, I can't seem to figure out how to authenticate via Python in the requests library. I'm open to any manner of authenticating, and have even tried using cookies from my browser per other suggestions on Stack Overflow, but I'm very unfamiliar with that method.
Am I completely missing something here? Most of these methods of authenticating I've found as solutions in Stack Overflow that seem to work for others.
s = requests.Session()
payload = {'usernameOrEmail': 'XXXXXXX', 'password':'XXXXXXX'}
s.post('https://XXXXXXX', json=payload, headers={'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36'})
s = requests.Session()
payload = {'usernameOrEmail': 'XXXXXXX', 'password':'XXXXXXX'}
s.post('https://XXXXXXX', json=payload)
url = 'https://XXXXXXX'
values = {'usernameOrEmail': 'XXXXXXX',
'password': 'XXXXXXX'}
r = requests.post(url, data=values, verify=False)
I've tried using the URL of the login page, the member overview page that you are forwarded to after logging in, the API URL's/endpoints, etc. I'm 98% sure I have the name value pairs correct for username and password.
I've even tried a simple get request and appending the usernameOrEmail and password fields on the URL.
I've gotten 401's, 404's, and a 405 most recently depending on the URL method combination.
Apologies in advance as I'm sure this is something extremely basic. Am I on the right track with submitting Username and Password or do I need to go the route of using browser cookies?
I'm using Spyder through Anaconda.
Thanks
Updated for Code Producing a 404:
import requests
s = requests.Session()
payload = {'username_or_email': 'XXXXXXX', 'password':'XXXXXXX'}
test=s.post('https://members.onepeloton.com/login', json=payload)
print(test)
There's nothing HTTP related you can't do in Python that you can do in your browser, except for running Javascript.
You're probably missing a request or some parameter. Maybe there has been a cookie set from before you started you browser session.
Try recording and emulating all the requests from an incognito tab, that way you're sure you start with the same state as your Python session.
Maybe we can help you more if you can tell us which website you're trying to authenticate to.
Firstly, there are already questions to this topic, but none cover up my problem, entirely, because it's either not the data I need or it's not working properly.
There are services like InstaDP that are able to show you the HQ version of any profile picture from Instagram. Now, I wonder how this is possible?
I did some research and were able to find a higher quality URL when accessing https://www.instagram.com/instagramforbusiness/?__a=1 (see profile_pic_url_hd, answered here). However, InstaDP seems to have a backend that returns a different url that redirects to a way higher quality image: https://instadp-cors-222621.appspot.com/get-hd?id=1107766105 (see at hd_profile_pic_url_info, I extracted the ID for the URL from the result of the ?__a=1 link). I tested this with my personal profile and was able to get the image of myself in an outstanding quality of 1024x1024. However, the ?__a=1 link seem only to return a link for my profile picture in 320x320.
Since InstaDP seem to not be the only player who is able to fetch HQ profile pictures I went ahead and compared the backends of those players. It seems that each service seem to have a different URL to the HQ profile picture of the same Instagram account. So my conclusion is that the Instagram API is involved in all that.
So I created a client key at https://www.instagram.com/developer/. I was also able to get my auth token and determine my logged in csrftoken for the X-CSRFToken header. Now my question is how to continue?
I found a few answers to this topic stating I should request https://i.instagram.com/api/v1/users/1107766105/info/, but it always returns the login page as HTML.
I tried a REST client that uses my Chrome cookies and logged into Instagram before, I tried to set my HTTP headers to X-CSRFToken:<mycookietoken> and Content-Type:application/json. (If I don't set the CSRFToken it errors, so I need to add it, but if the header is set I get the HTML again, even when the CSRFToken is correct. I don't get an error when the CSRFToken is wrong.)
I also tried setting the Origin, Referer and Host to trick Instagram in believing the request came from its own window location, without luck. Setting the Host will even cause a 400 bad request. Even adding my access token in the URL had no effect (?access-token=########).
To sum my question up, how do those services obtain the profile pictures in a such a great quality of up to 1024x1024 from the cdninstagram servers?
Although I am late, but this might be of someone's help:
Step 1:
First thing you need to get HD Instagram profile picture is their profile ID. This can be found in the source code of user profile link. For example if you view source of the following link https://www.instagram.com/abdulhaq0/ and search for "logging_page_id" you will get "profilePage_1285389476". The numbers following the profilePage are the ID for this account.
Step 2:
Next you need put ID in the following URL https://i.instagram.com/api/v1/users/{ProfileID}/info/ and open it in browser. In our case link would be https://i.instagram.com/api/v1/users/1285389476/info/
Step 3:
Now on the link above search for "hd_profile_pic_url_info". There you can get the URL of HD Instagram profile picture.
Hope this helped.
Steps:
1. Get the instagram post link. Eg : https://www.instagram.com/p/Bo-Jru-g7Wa/
or if you don't have the link, the instagram api provides you with a permalink option in the result array which for the above link is Bo-Jru-g7Wa
Now just follow add media?size=l after the url ie.,
Result: High quality image url:
https://www.instagram.com/p/Bo-Jru-g7Wa/media?size=l
you can see it in action here: https://jsfiddle.net/nmj1z7wo/fiddle URL
This link can be considered as a shorthand code to instagram image URL's which are very much bigger
I believe those sites like https://instadp.site/ that show the hi-res of the user profile image do not use the official Instagram API.
In the past it was possible to hack Instagram's CDN URLs to change parameters and get the high resolution from them, but nowadays the URLs are signed and if you change any parameters the URL will fail.
So, the only solution they may be using is to emulate a client. There is a popular PHP client for this: https://github.com/mgp25/Instagram-API
The instagram API have updated. Now we will get only the 320x320 sized image from
https://www.instagram.com/{username}/?__a=1
Even if you get the user id from this endpoint the "user info" endpoint does not return the hd url in its response and should pass a header too now.
import requests
def get_user_by_user_id(user_id):
if user_id:
base_url = "https://i.instagram.com/api/v1/users/{}/info/"
headers = {
'user-agent':'Mozilla/5.0 (iPhone; CPU iPhone OS 12_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 Instagram 105.0.0.11.118 (iPhone11,8; iOS 12_3_1; en_US; en-US; scale=2.00; 828x1792; 165586599)'
}
try:
res = requests.get(base_url.format(user_id),headers=headers)
print(res.json())
except Exception as e:
print("getting user failed, due to '{}'".format(e.message))
get_user_by_user_id(userid)
You can check this code and try. Websites like instadp I guess they does not use instagram's official API
The easiest and most reliable way of getting HD (1080x1080) profile pic is Instaloader CLI.
instaloader USERNAME_OF_INTEREST --no-posts --login YOUR_USERNAME
I have a Actions on Google project that uses api.ai for its actions. This is working well and I can see request/responses appear on the google assistant interface (On mobiles and simulator)
One of my usecases for api.ai needs to broken into 2 parts, in that we have to inform the user that the processing has started and then inform them again once its completed (without them reprompting for the output).
Im trying for a way to inform the user who is using the Google assistant when the processing is completed, but have failed so far. Something like this
User: I would like to see if my loan request is approved
Google Assistant: Hold on, let me check and let u know .
.... (Makes a webservice call to the backend asynchronously)
.... After few seconds ...
.... Postback to google assistant from the webservice
Google Assistant: Thanks for holding, your request is approved.
Im not sure how to do the "postback to google assistant" call. I have tried to get the SessionId from the Api.AI call and then use that to make a event request , but that doesnt seem to send the response to the assistant. Google Assistant seems to be using the formats defined in https://developers.google.com/actions/reference/rest/Shared.Types/AppRequest, but Im unsure how to get the ConversationToken and use that for sending the response back to the user.
Short answer: you can't do that.
Slightly longer answer: At least right now, there is no good way to send a notification. Your Action can only respond to a specific statement from the user. You can say something like "ask again in a minute and I should have a result for you", but that isn't a great experience. At Google I/O 2017, they announced that notifications would be coming to the Google Home at some point... but gave neither a time frame nor any information about an API.
Long, but probably still unsatisfying answer: You can look into Transactions which let them initiate purchase or request of some sort and then "check out". Once they have checked out, you would confirm that a transaction is being processed with an OrderUpdates and then can send updates with the status of the "order". These status updates can turn into notifications or user's can query the state of the order at any time. Transactions don't require payment, so this may work depending on your needs.
However, there are a few things to note. This is still in developer preview, so things may change in the future. It also doesn't work on all surfaces where the Assistant runs, so while it does work on Assistant on phones, it does not work on the Google Home right now.
I have a problem which was discussed in an other thread but while the author said he solved it is - for me at least - quit unclear how he did it.
Other thread: Yahoo finance historical stock price power query returns 301 response
I use Power Query an Add In for Excel which allows different query's, one is to grab website content, which I automated for yahoo data. Unfortunately it seems something changed with the yahoo site index and I am not able to use my query anymore.
If I try to recreate the query (build it from scratch again) I get the error "HTTP 301". Even with the "normal" query feature of excel the yahoo data is not available anymore.
Hopefully someone is able to help me.
Best wishes,
Andreas
You should set header "user-agent" to emulate browser
For examble Google Chrome
let
url = "https://finance.yahoo.com/quote/AAL/history?p=AAL",
#"user-agent"="Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.86 Safari/537.36",
web = Web.Contents(url, [Headers=[#"user-agent"=#"user-agent"]]),
html = Web.Page(web),
Data0 = html{0}[Data]
in
Data0
I'd like to write detector of Sony Bravia TV internet browser. I knew it's Opera browser but I don't know exactly what's properties to detect it ? Anyone know what's Opera version or how can I know it's Sony TV browser ?
Thanks a lot
Detection method depends on whenever you're doing it on a server side (in which case you have to look at the extra headers that browser in tv send) or on a client side javascript, in which case you have to look at the navigator.userAgent property.
As to extra headers, the only information I could find is this example headers:
X-AV-Physical-Unit-Info: pa="BRAVIA KDL-46XBR9";,X-AV-Client-Info: av=5.0; cn="Sony Corporation"; mn="BRAVIA KDL-46XBR9"; mv="1.7";
As you can see above, tv identifies itself with extra X-AV-Physical-Unit-Info and X-AV-Client-info headers
As to client-side detection of tv browser, I found this post (refering to google tv but still...) in which you can see the content of a navigator.userAgent property on two devices, including Sony Bravia.
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.127 Large Screen Safari/533.4 GoogleTV/ 162671