I am new to api and a little confused with what is the cause of the issue. I am accessing api through 2 ways:
a. I open the api url through the chrome website and it displays the result.
b. When I try to access the same api through requests.get(url), I get 403 error.
Do you know if this is possible that there is security issue involved when you access through python?
I tried same code for other api links as well and they work perfectly fine except this link.
PS: I can't share the links.
It got fixed by adding headers:
requests.get(url, headers={'User-Agent':'Mozilla/5.0'})
Related
I am trying to scrape dynamic websites and was using Puppeteer with Node.js before I realized i can just fetch the website's API directly and not have to render stuff that I don't need. By looking in the "Network" tab of Chrome's developer tools I could find the exact endpoints that returns the data I need. It works for most of the sites I am trying to scrape, but for some, especially POST requests, the API returns a "403: Forbidden" error code.
The API returns a success if I do a fetch-request directly from the Chrome console. But as soon as I try from a different tab, Postman, or Node using node-fetch I get "403: Forbidden".
I have tried copying the exact headers that are sent naturally from the website, and I have tried explicitly setting the "origin" and "referer" headers to the website's address but to no avail.
Is this simply a security measure that is impossible to breach or is there a way to trick the API into thinking that the request is coming from their own website?
I'm testing the website which have request to optimizely api to do some checking.
It request to url like https://cdn.optimizely.com/datafiles/XXX.json I suppose that this site required secure network.
I tried to open the url in cypress chrome and I get this error
This page isn’t workingcdn.optimizely.com didn’t send any data.
ERR_EMPTY_RESPONSE
But when I tried with the same network in chrome, I get fine response.
I need to be able to load the url to test my site.
Is there any solution to this matter. Please advice.
The resource returns 403 status code, that most likely indicates you don't have sufficient rights to see it:
Your Chrome outside Cypress runs might be set up differently, might already have session cookies.
You most likely need to figure out how to log into some account on the site throught Cypress.
Since cypress will not load optimizely neither nor google-analytics or any . My work around solution is by using cy.intercept() function in before/beforeEach
The code looks something like
cy.intercept('https://cdn.optimizely.com/datafiles/XXX8.json', {
"version": "4"
}
Reference: cypress-example-recipes
I was trying to do some simple authorization for ameritrade's developer platform. I was attempting.
According to the platform, the Endpoint I need to access is is:
https://auth.tdameritrade.com/auth?response_type=code&redirect_uri={uri}&client_id={client_id}}%40AMER.OAUTHAP
https://developer.tdameritrade.com/content/simple-auth-local-apps
When looking at the client_id, for the dev application, I was noticing that they may actually be referencing the Applications, Consumer Key instead? So i did just that, but when attempting to query the information, it returns: A third-party application may be attempting to make unauthorized access to your account. The reason why i think it is the consumer key, is listed at: https://developer.tdameritrade.com/content/getting-started
So I ended up doing something like:
from urllib.parse import urlencode, quote_plus
url = "https://auth.tdameritrade.com/auth?response_type=code&redirect_uri={uri}&client_id={client_id}}%40AMER.OAUTHAP".format(
uri=urlencode("http://localhost", quote_via=quote_plus),
client_id="JHBDFGJH45OOUDFHGJKSDBNG" #Sample
)
I dont think this is because I am currently in a different country currently, I think that something else is wrong here.
It doesnt follow through with it, but instead returns a 400 error with that information. Im not sure whats wrong though.
This happens when you copied the callback URI incorrectly. Imagine if this were a client application, and TD detected that the application is trying to send the user to a different URL than the app is configured with. If they send the callback request to that application, it will receive the token and gain full control over your account.
Have you double and triple checked that you're copying the callback URL correctly, protocol name, ports, and trailing slashes and everything? Also, consider using an API library instead of writing your own. You can find documentation about this specific error here.
I had this issue and I solved it using simply using http://127.0.0.1 on the call back URI of the App.
I then used below URL and it worked as expected.
https://auth.tdameritrade.com/auth?response_type=code&redirect_uri=http%3A%2F%2F127.0.0.1&client_id={MyConsumerKey}%40AMER.OAUTHAP
Just in case anyone is still having this problem, make sure the callback URI is spelled EXACTLY the same as you specified when creating the app. I was having this problem because I set the callback on the TD developer website to "https://localhost/" and used "https://localhost" in the URL instead (missing the slash at the end). As soon as I added the slash at the end, it worked.
I found out that the issue is caused by the way the callback URL is set. It have to be exactly the same as the callback URL you have typed in at the apps details on the TD developer API page. I tried several permutations and indeed to get the authorization to work both have to be the same. eg. https or http.. end with '/' or does not, it matters. There is also no need to URL encode it.
has anyone seen this before so I am getting a 502 bad gateway error on my app, the issue I have is that the detailed error information I am getting says my requested url is https://SOX:80/api however my site is configured to use https://sox.domain.com and the site largely works pulling the various JS files required
my app service name is SOX in the azure dashboard so I assume that is where it is picking up SOX from but I have no idea why it is using this.
So overall the issue had me perplexed... however with more testing I soon figured out what was going on.
my backend is Dotnet core Azure throwing the 502 bad gateway was its way of handling exceptions ultimately the problem was code based.
I am mentioning this purely so that it will help others
my first issue was based on cert handling it seems dotnet runs in a container that is specified by your app name as i mentioned above https://SOX:80
the below was causing my issues
sslPolicyErrors = X509StoreStoreHelper.ValidateSSLPolicy(cert.Thumbprint, cert);
after commenting this out for testing my problem went away(we are putting in a proper fix )
my second issue came from using an unsupported view in Azure SQL master.sys.master_files which again just threw a 502 bad gateway error referencing https://SOX:80
please note I have used https://SOX:80 as a reference to mask the real site.
hope this helps the next person.
Based on your description, I have checked your site (https://sox.azurewebsites.net/) and found that it contains three static files (index.html,generic.html,elements.html). I viewed your website in Chrome incognito window as follows:
I did not find any requests against https://SOX:80/api in your html page or JavaScript files. Please try to access your website in a new incognito window to isolate the cache issue or just press CTRL + F5 to refresh your current page to narrow this issue. Moreover, you need to check whether you have configured URL Rewrite. If you still could not solve this issue, you need to update your question with the details for us to reproduce this issue.
I have some java code that retrieves blogs through the REST API's. I am not using the social business toolkit, but we have our own framework for that.
The application works perfectly on an on-premise connections environment and has worked on multiple versions.
However when switching to Connections Cloud, some parts stopped worked.
We get a 403 - Forbidden exception on 2 occasions:
Getting the details of a blog post: /blogs/[blog-id]/feed/entry/atom?entryid=[entry-id]
Getting images inside the blog post: /blogs/[blog-id]/resource/BLOGS_UPLOADED_IMAGES/[image file name]
I have fixed issue 1) by switching to the plublishing API: /blogs/[blog-id]/api/entries/[entry-id].
I cannot find a way to fix issue 2). I have also found 2 other image urls:
https://apps.ce.collabserv.com/blogs/[blog-id]/api/media/[file-name]
https://apps.ce.collabserv.com/blogs/[blog-id]/api/media/BLOGS_UPLOADED_IMAGES/[file-name].media
Both return:
<sp_0:error xmlns="http://incubator.apache.org/abdera" xmlns:sp_0="http://incubator.apache.org/abdera">
<code>404</code>
<message>Not Found</message>
</sp_0:error>
I want to authenticate by using Basic Authentication when possible. This does not appear to work with the given 403 urls.
My guess is that this the basic authentication header is not picked up. I have seen this before.
I used to fix this by first calling another URL that does support basic authentication and using the Ltpa cookies to authenticate the image url.
This also does not work: I do get LtpaTokens, but when I pass all the cookies to the URL, the image still does not work.
I prefer not to use OAuth of OAuth 2 at this moment. Is there any other way to fix this?
Anybody else managed to retrieve BLOGS_UPLOADED_IMAGES?
The issue is can also be reproduced in a browser.
Make sure you are not yet authenticated and the blog has posts with
images
Go to /blogs/[blog-id]/api/media
Authenticate using the popup in the browser The Atom feed now appears. This contains the images of your blog.
403 when opening:
/blogs/[blog-id]/resource/BLOGS_UPLOADED_IMAGES/[image]
404 xml when opening: /blogs/[blog-id]/api/media/* links