I am sending a https GET Type request to an external API.
First, I created the https agent as below:
import https from 'https';
const KeepAliveAgent = new https.Agent( {
keepAlive: true
} );
then set the request options as below:
let options = {
url: 'externalapiurl',
method: "GET",
qs: queryString,
agent: KeepAliveAgent
};
I just mentioned sample strings for url and qs, in the original request I am using the actual api url and querystring, then I am sending the request as below:
console.time( "requestTime" );
request( options, ( err, response, body ) => {
if ( err ) {
logger.warn( err.message );
}
console.timeEnd( "requestTime" );
});
This is working fine, but I am printing the time taken for response above and this time is much more when I send the request behind a proxy, when I doesn't use proxy it's taking less than half second but with proxy it's taking around 3 secs, so it seems the "keep alive" is not working behind the proxy, how to make this work?
I tried same request using https-proxy-agent node module, but still the issue persists, appreciate any help.
All about measuring performance of HTTP/HTTPS Proxies. https://blog.thousandeyes.com/measuring-performance-with-http-proxies/][1]
Code given by you seems correct.
I suggest use socketio in nodejs for such implementation you are doing.
Related
A few days ago I've been researching how I can listen for https requests in my browser and overwrite some data (headers, body) from the request before it reaches the final destination (site).
I tried many times. I came to use the modules
mockttp, web-sniffer, hoxy, among others.
As the application I'm developing is with selenium, I tried to create a proxy server with the modules mentioned above and make them go through my application before being "dispatched", however, none of them were successful.
Note: I was able to listen to requests, change their response, and other things. But I didn't get what I really wanted: change the request data before it was sent to the site. Something similar to the breakpoint functionality present in Fiddler, Charles, HTTP Tool Kit and others applications.
What do I really hope?
I'd like to hear my browser's requests -> pause HTTPS requests (breakpoint), change something anyway -> forward/follow/dispatch with/the request.
One of the failed attempts:
The new header is not really inserted in the request! (browser was configured with proxy server and I could see the requests being shown in the console)
import mockttp from "mockttp";
import fs from "fs";
import * as Sniffer from "web-proxy-sniffer";
(async () => {
// Create a proxy server with a self-signed HTTPS CA certificate:
const https = await mockttp.generateCACertificate();
fs.writeFile("key.pem", https.key, console.log);
fs.writeFile("cert.pem", https.cert, console.log);
const proxy = Sniffer.createServer({
certAuthority: {
key: fs.readFileSync(`./key.pem`),
cert: fs.readFileSync(`./cert.pem`),
},
});
proxy.intercept(
{
// Intercept before the request is sent
phase: "request",
},
(request, response) => {
request.headers.test = "a simple test";
return request;
}
);
proxy.listen(8001);
})();
Would this be possible with NodeJS?
(Sorry for this "rude" english)
I have a node.js GET API endpoint that calls some backend services to get data.
app.get('/request_backend_data', function(req, res) {
---------------------
}
When there is a delay getting a response back from the backend services, this endpoint(request_backend_data) is getting triggered exactly after 2 minutes. I have checked my application code, but there is no retry logic written anywhere when there is a delay.
Does node.js API endpoint gets called twice in any case(like delay or timeout)?
There might be a few reasons:
some chrome extensions might cause bugs. Those chrome extensions have been causing a lot of issues recently. run your app on a different browser. If there is no issue, that means it is chrome-specific problem.
express might be making requests for favicon.ico. In order to prevent this, use this module : https://www.npmjs.com/package/serve-favicon
add CORS policy. Your server might sending preflight requests Use this npm package: https://www.npmjs.com/package/cors
No there is no default timeouts in nodejs or something like that.
Look for issue at your frontend part:
can be javascript fetch api with 'retry' option set
can be messed up RxJS operators chain which emits events implicitly and triggers another one REST request
can be entire page reload on timeout which leads to retrieve all neccessary data from backend
can be request interceptors (in axios, angular etc) which modify something and re-send
... many potential reasons, but not in backend (nodejs) for sure
Just make simple example and invoke your nodejs 'request_backend_data' endpoint with axois or xmlhttprequest - you will see that problem is not at backend part.
Try checking the api call with the code below, which includes follwing redirects. Add headers as needed (ie, 'Authorization': 'bearer dhqsdkhqd...etc'
var https = require('follow-redirects').https;
var fs = require('fs');
var options = {
'method': 'GET',
'hostname': 'foo.com',
'path': '/request_backend_data',
'headers': {
},
'maxRedirects': 20
};
var req = https.request(options, function (res) {
var chunks = [];
res.on("data", function (chunk) {
chunks.push(chunk);
});
res.on("end", function (chunk) {
var body = Buffer.concat(chunks);
console.log(body.toString());
});
res.on("error", function (error) {
console.error(error);
});
});
req.end();
Paste into a file called test.js then run with node test.js.
Basic problem: you've followed the tutorial, you've fired up the Hapi server, it's running... but it doesn't work. A direct call via curl will get something, using a web browser to directly load the API call will get something... but using that API endpoint within an app, say, Angular or React, and it bombs out with an error message like:
Access to XMLHttpRequest at 'https://localhost:3000/server/ping' from origin 'http://localhost:5000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
And it's true: you check the headers, and Access-Control-Allow-Origin is not on that list at all. So your app, having gotten blocked here in the preflight request, isn't even going to make the actual GET/POST call.
Here's the full file of a fully working Hapi v20.2.0 server, in TypeScript:
'use strict'
import * as fs from 'fs'
import * as util from 'util'
import * as path from 'path'
import * as os from 'os'
import * as Hapi from '#hapi/hapi'
import * as Http2 from 'http2'
const strFullNameCert:string=path.resolve(
os.homedir(),
'ssl',
'domain.crt')
const strFullNameKey:string=path.resolve(
os.homedir(),
'ssl',
'domain.key')
const key :Buffer = fs.readFileSync(strFullNameKey)
const cert:Buffer = fs.readFileSync(strFullNameCert)
const sslDetails ={key,cert}
const server = new Hapi.server({
listener: Http2.createSecureServer(sslDetails), // optional: remove this line for http1.1
host: 'localhost',
port: 3000,
tls: sslDetails,
routes: {
cors: true
},
})
console.log(`Host : ${server.info.host}`)
console.log(`Port : ${server.info.port}`)
console.log(`Method : ${server.info.protocol}`)
console.log(`Hapi : v${server.version}`)
server.route({
method: 'GET',
path:'/server/ping',
handler: async (request, reply) => {
console.log(`>>>ROUTE>>> : ${request.route.path}`);
const response = reply.response({ qSuccess: true, qResult: 'pong' })
return response;
}
})
server.start()
To reiterate, this code will "work", it will serve up a response if you load the /server/ping route in an independent way. If you were building a web server to serve pages, this would likely be sufficient to get going.
This code will still fail CORS validation in a web app. Why? Because the request to /server/ping is never even going to be made. The app will send a preflight OPTIONS request first. And there's nothing in this code to handle that. So nothing you do in the server.route area, messing with route options, or adding headers, is going to fix this. Ten jillion different setups in the main server instantiation of routes:cors wont fix this, because they also don't address the actual problem.
I added these, at the top of my middleware, to respond to options(pre-flight) request.
Might cause problems in other areas of the app that use the options, but worked for my case/issue.
async function(request: Request, h: ResponseToolkit) {
if (request.method === "options") {
const response = h.response({});
response.header("Access-Control-Allow-Origin", "*");
response.header("Access-Control-Allow-Headers", "*");
return h.response(response).takeover();
}
// more checks....
},
The problem is there isn't a route set up that's dealing with the OPTIONS request that Chrome/Firefox will send before they attempt the GET/POST to the API being served up by Hapi. Add this code to the above file, just above server.start()
server.route({
method : 'OPTIONS',
path: '/{any*}',
handler : async (request, reply) => {
console.log(`======options: ${request.route.path}`)
const response = reply.response({})
response.header('Access-Control-Allow-Origin', '*')
response.header('Access-Control-Allow-Headers', '*')
return response;
}
})
Now you'll see that when you attempt to use the API from your app, it's going to make two calls: first, the options call, and then if that passes (and with these headers it now will) it will make the actual call you're interested in.
The "cors:true" route option, it really feels like it's going to "enable cors". I mean, yes... but really, no. Think of it more like it permits Hapi to do cors... you still have to do the work of "handling cors".
I am using request npm module in my app, to make to create a http client, as this.
var request = require('request');
And each time, I make a request to some server, I pass the options as below:
var options = {
url: "whateverurl...",
body: { some json data for POST ... }
}
request(options, cb(e, r, body) {
// handle response here...
})
This was working fine, until I started testing with high load, and I started getting errors indicating no address available (EADDRNOTAVAIL). It looks like I am running out of ephemeral ports, as there is no pooling or keep-alive enabled.
After that, I changed it to this:
var options = {
url: "whateverurl...",
body: { some json data for POST ... },
forever: true
}
request(options, cb(e, r, body) {
// handle response here...
})
(Note the option (forever:true)
I tried looking up request module's documentation about how to set keep-alive. According to the documentation and this stackoverflow thread, I am supposed to add {forever:true} to my options.
It didn't seem to work for me, because when I checked the tcpdump, the sever was still closing the connection. So, my question is:
Am I doing something wrong here?
Should I not be setting a global option to request module, while I am "require"ing it, instead of telling it to use {forever:true}, each time I make a http request? This is confusing to me.
Im trying to use node.js to send http requst and call baidu map API.
my code in blow:
If you past the url and use browser directly, it will give right response in right format.
But when I use node to send request, I get problem.
var request = require('request');
request(
{ method: 'GET',
uri: 'http://api.map.baidu.com/place/v2/suggestion?query=beijing®ion=131&output=json&ak=****hLQKu9ap9fPq5N1ExF1Kk7xe5Eah'
}
, function (error, response, body) {
res.json({
res:response
})
}
)
Meanwhile, if I change the url contains some Chinese like:
http://api.map.baidu.com/place/v2/suggestion?query=北京理工大学®ion=北京&output=json&ak=****hLQKu9ap9fPq5N1ExF1Kk7xe5Eah
In node.js it will give status code 400 and totally wrong response.
you must encode your uri with encodeURI
uri: encodeURI('http://api.map.baidu.com/place/v2/suggestion?query=北京理工大学®ion=北京&output=json&ak=3104hLQKu9ap9fPq5N1ExF1Kk7xe5Eah')