Warning: file_get_contents(https://www.instagram.com/p/CbNmjMuKz8r/?__a=1): failed to open stream: HTTP request failed! HTTP/1.1 429 Too Many Requests
enter code here if($image->getLink()!='')
{
$insta_url=$image->getLink().'?__a=1';
}
else
{
$insta_url='https://www.instagram.com/p/'.$_POST['Id'].'/?__a=1';
}
$b1 = json_decode(file_get_contents($insta_url), TRUE);
Related
I am using http-filter plugin in filter-stage
filter {
http {
url => "someUrl"
verb => "GET"
target_body => "apiResponse"
target_headers => "apiResponseHeader"
}
}
and I want to send the result of http filter to kafka, but when http filter got error, it gives me error as below
(below error is intentionally caused: 404)
[2023-02-09T14:47:34,517][ERROR][logstash.filters.http ][main][45dd1acfcf38ba5088de03ee36672bc4b8f8046ff3853965393fa1c6b80f4cb8] error during HTTP request {:url=>"baseUrl", :code=>404, :response=>"{\"timestamp\":\"2023-02-09T14:47:34.507+00:00\",\"status\":404,\"error\":\"Not Found\",\"path\":\"someWrongUrl"}"
now, i want to re-process(parse) above error to
url: baseUrl
code: 404
is there any way to re-process the output or catch the error message in filter-stage?
thanks
I have a simple Web server, which should send a file. I took the code from another answer.
#! /usr/bin/node
const FS = require ('fs');
const HTTP = require ('http');
const server = HTTP.createServer ();
server.on ('request', (request, response) => {
switch (request.url) {
case '/':
switch (request.method) {
case 'GET':
console.log ("GET /");
let stat = FS.statSync ('index.html');
console.log (stat.size);
response.writeHead (200, { 'Content-Type': 'text/html',
'Content-Lenght': stat.size });
let index = FS.createReadStream ('index.html', 'UTF-8');
index.pipe (response);
response.end ();
return;
}
break;
}
response.writeHead (400, {});
response.end ();
});
server.listen (8080);
When I try to send a GET request with curl, I get no content. My server reports, that the index.html file has 324 bytes:
$ ./server.js
GET /
324
But curl does not show the content. There header contains the content length, but the body is missing.
$ curl -v --noproxy \* http://localhost:8080/
[...]
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html
< Content-Lenght: 324
< Date: Sat, 21 Nov 2020 19:24:31 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact
I looks as if the connection got closed before the file has been piped. Is the the error and how can I avoid it?
Remove the response.end ();. You're prematurely closing the response BEFORE the .pipe() gets to do its work (it's asynchronous so it finishes over time and returns before it's done).
In the default configuration, .pipe() will end your response for you when it's done.
You will also notice that the other answer you took this idea from did not have a response.end().
In NodeJS and express I am handling a batch request to a OData API. Sometimes the OData service returns invalid results in the response which is denoted by a status 500 (could be more than one, since this is a OData batch response), even though the request header status code returns 200 because the connection to the API itself was successful.
I am new to NodeJS and am not sure how / the best way to detect these status 500 in the response body so that I can retry the request (reason is these 500 errors are usually timeouts and on the next retry goes away).
Do I need a package to be able to retrieve or intercept res.body? I don't see it available in express.
Request and response example
Request header
Request URL: https://odata/$batch
Request Method: POST
Status Code: 202 Accepted
Remote Address: xxxx
Referrer Policy: no-referrer-when-downgrade
Response
--batch_1564714041997_0
Content-Type: application/http
Content-Transfer-Encoding: binary
HTTP/1.1 500 OK
Content-Type: text/plain;charset=utf-8
Internal server error
--batch_1564714041997_0
Content-Type: application/http
Content-Transfer-Encoding: binary
HTTP/1.1 500 OK
Content-Type: text/plain;charset=utf-8
Internal server error
--batch_1564714041997_0--
I would use a regular expression and a capturing group for the HTTP status code.
var regExForHTTPStatusCodeDetection = RegExp('HTTP\/1\.\\d (\\d{3})','g');
var requestNumber = 1;
while ((regExMatches = regExForHTTPStatusCodeDetection.exec(res.body)) !== null) {
// Get status code from first capturing group
var httpStatusCode = regExMatches[1];
// Output for above example:
// "Request 1 has returned HTTP 500."
// "Request 2 has returned HTTP 500."
console.log(`Request ${requestNumber} has returned HTTP ${}.`);
requestNumber++;
}
Cannot save file larger than pipe chunk size (~64K).
Using mongodb 3.4.0, Relevant node dependencies
restify 4.2.0
mongodb ^2.2.12
lodash 4.16.6
bookeeppingData = {request, id, ...meta}
clone = lodash.cloneDeep(bookkeepingData)
const {
request: req,
id: _id,
meta: metadata,
} = clone
const bucket = new mongodb.GridFSBucket(
db,
{bucketName: 'my_gridfs_collection'}
);
const uploadSteam = bucket.openUploadStreamWithId(
_id,
undefined,
{metadata}
);
req.on('data', (chunk) => {
console.log(`Received ${chunk.length} bytes of data.`);
});
req.on('end', () => {
console.log('There will be no more data.');
});
req.on('error', (e) => {
console.log('req on error', e);
});
uploadSteam.on('finish', function() {
console.log('finsish');
keepThebooks(bookkeepingData);
});
uploadSteam.on('error', (e) => {
console.log('uploadstream on error', e);
});
req.pipe(uploadSteam);
}
When sending a file smaller than the ~64K, The console output is
Received 57259 bytes of data.
There will be no more data.
finish
This is the corresponding curl --verbose output (minus the json response):
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8060 (#0)
* Server auth using Basic with user 'driver'
> POST /artifact?branch=xyz&role=PHOTO&where.lat=55&where.long=77.2&where.acc=12&sys=system&id=1234&sys=system2&id=1234b&created=2018-11-01T18:41:50.850Z&tz=-600 HTTP/1.1
> Host: localhost:8060
> Authorization: Basic xxxxxxxxx
> User-Agent: curl/7.61.1
> Accept: */*
> Content-Type: image/png
> Content-Length: 57259
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Cache-Control: no-cache, no-store, must-revalidate
< Content-Type: application/json
< Content-Length: 388
< Date: Wed, 23 Jan 2019 20:58:16 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
When sending a file larger than ~64K, the console output is:
Received 65536 bytes of data.
(That's it -- no error)
The corresponding curl --verbose output is:
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8060 (#0)
* Server auth using Basic with user 'driver'
> POST /artifact?branch=xyz&role=PHOTO&where.lat=55&where.long=77.2&where.acc=12&sys=system&id=1234&sys=system2&id=1234b&created=2018-11-01T18:41:50.850Z&tz=-600 HTTP/1.1
> Host: localhost:8060
> Authorization: Basic xxxxxxxxx
> User-Agent: curl/7.61.1
> Accept: */*
> Content-Type: image/png
> Content-Length: 84801
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
I expected this to work, and for the console to be something like:
Received 65536 bytes of data.
Received 19274 bytes of data.
There will be no more data.
finish
I am on the same team.
Turns out we were deep cloning the object containing the stream. When we stopped cloning the entire file uploads fine.
I'm using node-twilio and I keep getting a "Error: Unable to reach host: "api.twilio.com" for every request. We've checked the packets via mtr and they are reaching api.twilio.com. Running on debian on GCE.
After days of digging around, found out that the node-twilio module shows many errors incorrectly as:
"Error: Unable to reach host: "api.twilio.com".
The following lines:
var error = null;
if (err || (response && (response.statusCode < 200 || response.statusCode > 206))) {
error = {};
// response is null if server is unreachable
if (response) {
error.status = response.statusCode;
error.message = data ? data.message : 'Unable to complete HTTP request';
error.code = data && data.code;
error.moreInfo = data && data.more_info;
} else {
error.status = err.code;
error.message = 'Unable to reach host: "'+client.host+'"';
}
}
This happens because you have a self signed certificate in your chain and the underlying module twilio depends on is request, which is throwing the following error:
Error: SELF_SIGNED_CERT_IN_CHAIN but this is not the error being thrown by node-twilio (bad error propagation on their part)
There are 2 fixes:
1.Tell nodejs to ignore self signed certificates in chain by setting:
export NODE_TLS_REJECT_UNAUTHORIZED=0
Find the self signed certificate and remove it from the chain. Here is an example using openssl: https://serverfault.com/questions/590870/how-to-view-all-ssl-certificates-in-a-bundle
References:
https://github.com/request/request
https://github.com/twilio/twilio-node/blob/45858420688854494c2ed476a1997773c33a32a0/lib/Client.js
Ignore invalid self-signed ssl certificate in node.js with https.request?
It may be because of your internet connection.
After couple of minutes, if you have internet, try again and it should work.