I'm trying to fetch a page from IMDB, but for some odd reason it gives me error 400 when I'm using request-promise
But the same query works fine if I'm using CURL:
curl "https://www.imdb.com/title/tt6306064/mediaviewer/rm3146075904"
My nodejs code:
async function getMoviePosterImage(mediaViewerUrl) {
const options = {
uri: mediaViewerUrl
};
try {
const mediaViewerHtml = await request.get(options);
return mediaViewerHtml;
} catch (error) {
console.error(error.statusCode);
}
}
await getMoviePosterImage(
"https://www.imdb.com/title/tt6306064/mediaviewer/rm3146075904"
);
Things I have tried so far:
Setting a user agent
Setting keep-alive
Having cookie jar enabled
On my end, I just tried locally and the same error happened for me. Not a definitive answer, but I have a feeling that IMDB protects against web scrapers.
Related
I have tried looking at other articles on SO related to this question, but I am unable to resolve this issue.
I have a graphql api and in my component on the UI side, I am calling it as such
useEffect(()=>{
const leaders = graphQLClient.request(GetLeadersQuery)
if(leaders && leaders.leaders) {
let leadersList = JSON.parse(leaders.leaders)
setLeaders(leadersList)
}
}, [leaders])
On the server side, this looks like this
leaders: (_parent, args, _context) => {
return (JSON.stringify({
name:"Jane Dooe"
}))
This is the query that I am using
query {
leaders
}
Now everytime, my client side code is hit, I see the following error in my terminal.
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
My use case is very simple and yet , I am unable to see why I am encountering this error.
Update
On my client side, I tried doing the following, just to make sure that I am resolving the promises, but I am still getting the same result.
useEffect(()=>{
async function getLeaders(){
const leaders = await graphQLClient.request(GetLeadersQuery)
console.log({ leaders })
if (leaders && leaders.leaders) {
let leadersList = JSON.parse(leaders.leaders)
setLeaders(leadersList)
}
}
getLeaders()
}, [])
On the server side I am using
NextJS
apollo-server-micro
On the client side I am using
React
graphql-request
Can you please help ?
Thanks
What I have:
I adapted this example from Microsoft docs.
// functions.js
/**
* Get data
* #customfunction
* #returns {string[][]}
*/
async function getData() {
try {
const url = "https://api.example.com/some/objects/";
const token = await OfficeRuntime.storage.getItem("Token");
const authString = `Token ${token.toString()}`;
const response = await fetch(url, {
headers: { Authorization: authString }
});
if (!response.ok) {
throw new Error(response.statusText);
}
const jsonResponse = await response.json();
return jsonResponse.map(obj => {return [obj.id.toString(), obj.name]};
} catch (error) {
return [["ERROR", error.message]];
}
}
Added api.example.com to <AppDomains>
Item "Token" is present in OfficeRuntime.storage
Same API call with Postman works fine
The Add-In is not served from localhost (because of CORS reasons etc.)
What I get:
Because it is not developed locally it is very hard to debug ui-less custom functions. Therefore the only visible error I get so far, is the one I receive and return to Excel in the catch-block. It is an unhelpful error message: Network request failed
Can anyone help me with any suggestions?
Reason why it did not work was a known issue that custom functions run in a separate JavaScript runtime as described here. This runtime allows only CORS-safelisted request header because it lacks of CORS-Preflight -> therefore the Network request failed error with the Authorization header.
How I solved it:
Configure the Excel-Add-in to use a shared JavaScript runtime as described here.
Add <script src="functions.js"></script> just before the <\head> element in taskpane.html as described here.
Build the project npm run build
Clear the cache as described in the first two steps here.
Run Excel and load your Add-in.
Describe the issue
I'm not really sure if this is an Axios issue or not. The following code runs successfully on my local development machine but always time out whenever I run it from the cloud (e.g. AWS Lambda). Same thing happens when I run on repl.it.
I can confirm that AWS Lambda has internet access and it works for any other API but this:
https://www.target.com.au/ws-api/v1/target/products/search?category=W95362
Example Code
https://repl.it/repls/AdeptFluidSpreadsheet
const axios = require('axios');
const handler = async () => {
const url = 'https://www.target.com.au/ws-api/v1/target/products/search?category=W95362';
const response = await axios.get(url, { timeout: 10000 });
console.log(response.data.data.productDataList);
}
handler();
Environment
Axios Version: 0.19.2
Runtime: nodejs12x
Update 1
I tried the native require('https') and it times out on both localhost and cloud server. Please find sample code here: https://repl.it/repls/TerribleViolentVolume
const https = require('https');
const url = 'https://www.target.com.au/ws-api/v1/target/products/search?category=W95362';
https.get(url, res => {
var body = '';
res.on('data', chunk => {
body += chunk;
});
res.on('end', () => {
var response = JSON.parse(body);
console.log("Got a response: ", response);
});
}).on('error', e => {
console.log("Got an error: ", e);
});
Again, I can confirm that same code works on any other API.
Update 2
I suspect that this is something server side as it also behaves very weirdly with curl.
curl from local -> 403 access denied
curl from local with User-Agent header -> success
curl from cloud server -> 403 access denied
It must be server side validation, something related to AkamaiGHost.
You have probably placed your Lambda function in a VPC without Internet access to the outside world. Try check the VPC section in your lambda configuration, and setup an internet gateway accordingly
You should try by wrapping axios call into try/catch maybe that will catch the issue.
const axios = require('axios');
const handler = async () => {
try {
const url = 'https://www.target.com.au/ws-api/v1/target/products/search?category=W95362';
const response = await axios.get(url, { timeout: 10000 });
console.log(typeof (response));
console.log(response);
} catch (e) {
console.log(e, "error api call");
}
}
handler();
As suggested by Akshay you can use try and catch block to get the error. Maybe it helps you out.
Have you configured Error Handling for Asynchronous Invocation?
To configure error handling follow the below steps:
Open the Lambda console Functions page.
Choose a function.
Under Asynchronous invocation, choose Edit.
Configure the following settings.
Maximum age of event – The maximum amount of time Lambda retains an event in the asynchronous event queue, up to 6 hours.
Retry attempts – The number of times Lambda retries when the function returns an error, between 0 and 2.
Choose Save.
axios is only Promise based HTTP client for the browser and node.js and as you set timeout: 10000 so I believe timeout issue is not from its end.
Although your API
https://www.target.com.au/ws-api/v1/target/products/search?category=W95362
is working fine on the browser and rendering JSON data.
and Function timeout of lambda is by default 15 minutes, which I believe is enough for the response. There may be another issue.
Make sure you have set other configurations like permissions etc. as suggested in the documentation.
Here you can check the default limits for AWS lambda.
I am using Nodejs to write a simple web app that needs to read content from readme files using the GH api.
Everything works, but I am can't sort how to create requests as an authenticated user.
Sorry bit of a noob here :-) but is it not enough to add my client and secret key or an access_token as a parameter to my url? I have tried both and both seem to time out after 60 requests instead the 5000 the docs say.
I have looked at this site Github Rate Limits but I think I have done what it says.
Do I need to add a token on my server? Like how public and private ssh keys work? - Sorry, just trying to get an understanding of this.
This worked for me recently for getting a list of issues from Github. Nothing else set up on server or similar.
I used a token created with https://github.com/settings/tokens/new
const chalk = require("chalk");
const sa = require("superagent");
const { getProperty } = require("../context");
async function getIssues(org) {
try {
const url = `https://api.github.com/orgs/${org}/issues?state=open`;
const apiToken = await getProperty("github.token");
const res = await sa
.get(url)
.set("Authorization", `token ${apiToken}`)
.send();
res.body.forEach(issue => {
console.log(issue.title);
});
} catch (err) {
console.error(err);
}
}
module.exports = getIssues;
I'm trying to get a webpage via node https.request(). Doing so results in an error getting logged by my code. Using the node request module has the same result:
problem with request: 140398870042432:error:140773F2:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert unexpected message:s23_clnt.c:658:
The following indicates the wrong SSL version is being used, but I cannot find a way to change the version: curl error: "sslv3 alert unexpected message". Using curl from my terminal returns a response as does hitting the URL in my browser (it is a login page). My code is below.
var request = require('request')
request.get("https://icmserver.wit.ie/balance", function(err, res, body) {
if (err) {
return console.log(err)
}
return body;
});
Does anyone have any idea what might be happening here?
Try to use options = { secureProtocol: 'SSLv3_method' } in the request you are making.
We hit the same problem. By default, request uses the https.globalAgent. So we added the code near the top of our script.
var https = require('https');
https.globalAgent.options.secureProtocol = 'SSLv3_method';
All of a sudden everything worked.
In case website uses ECDH curve, for me the issue resolved only by adding this option:
request({ url, agentOptions: {
ecdhCurve: 'P-521:P-384:P-256',
},(err,res,body) => {
JFYI, May be this will help someone.