I think I'm missing something about http and https requests
I have a variable that contains a URL, for example:
http(s)://website.com/a/b/file.html
I would like to know if there's a easy way to make a request to that URI to get the data
To make a http(s)Request, here's what I have to do now:
Test if the URL is http or https to make the appropriate request
Remove the http(s):// part and put the result in a variable (If I specify http or https in the hostname, I get an error)
Separate the hostname from the path: website.com and `/a/b/file.html
Put this variables in the options objects
Is this a must or are they easier solutions that don't involve getting out the hostname and path, and testing if the site is in http or https ?
Edit: I can't use http.get as I need to put some specific options
In order to get all components out of URL you need to parse it. Node v0.10.13 has stable module for it: url.parse
This is simple example how to do so:
var q = url.parse(urlStr, true);
var protocol = (q.protocol == "http") ? require('http') : require('https');
let options = {
path: q.pathname,
host: q.hostname,
port: q.port,
};
protocol.get(options, (res) => {...
For those ending up here, protocol includes :, and pathname does not include search so it must be added manually. Parameters shouldn't be parsed as they are not needed (so you can save computing time :)
Also it's not really a best practice to require inside a function and probably this code will end up inside a function so having all this improvements, so I would rewrite the answer to something like this:
import * as url from 'url';
import * as https from 'https';
import * as http from 'http';
const uri = url.parse(urlStr);
const { request } = uri.protocol === 'https:' ? https : http;
const opts = {
headers, // Should be defined somewhere...
method: 'GET',
hostname: uri.hostname,
port: uri.port,
path: `${uri.pathname}${uri.search}`,
protocol: uri.protocol,
};
const req = request(opts, (resp) => { ...
Related
In order to properly build my urls in my xml sitemaps and rss feeds I want to determine if the webpage is currently served over http or https, so it also works locally in development.
export default function handler(req, res) {
const host = req.headers.host;
const proto = req.connection.encrypted ? "https" : "http";
//construct url for xml sitemaps
}
With above code however also on Vercel it still shows as being served over http. I would expect it to run as https. Is there a better way to figure out http vs https?
As Next.js api routes run behind a proxy which is offloading to http the protocol is http.
By changing the code to the following I was able to first check at what protocol the proxy runs.
const proto = req.headers["x-forwarded-proto"];
However this will break the thing in development where you are not running behind a proxy, or a different way of deploying the solution that might also not involve a proxy. To support both use cases I eventually ended up with the following code.
const proto =
req.headers["x-forwarded-proto"] || req.connection.encrypted
? "https"
: "http";
Whenever the x-forwarded-proto header is not present (undefined) we fall back to req.connection.encrypted to determine if we should serve on http vs https.
Now it works on localhost as well a Vercel deployment.
my solution:
export const getServerSideProps: GetServerSideProps = async (context: any) => {
// Fetch data from external API
const reqUrl = context.req.headers["referer"];
const url = new URL(reqUrl);
console.log('====================================');
console.log(url.protocol); // http
console.log('====================================');
// const res = await fetch(`${origin}/api/projets`)
// const data = await res.json()
// Pass data to the page via props
return { props: { data } }
}
I was looking through my codebase today, the portion which sets up the server and found the following lines:
var https = require('https');
https.globalAgent.options.secureProtocol = 'TLSv1_2_method';
function createHttpsServer(app) {
var https = require('https');
var fs = require('fs');
const options = {
secureProtocol: 'TLSv1_2_method',
// ...
};
var server = https.createServer(options, app);
return server;
}
It looked like code duplication to me and I am not sure why these do different things (or do they?).
A colleague of mine told me that the top one is for controlling TLS in HTTPS requests made from NodeJS, which in turn, gives us access to the https.agent which is used for all things related to client HTTP requests.
This was also compared to the ServicePointManager in the .NET world.
So do these methods both do different things? At some point, our code does:
var server = protocol === 'https' ? createHttpsServer(app) : createHttpServer(app);
Wouldn't that be using the same server at the end of the day?
var server = protocol === 'https' ? createHttpsServer(app) : createHttpServer(app);
The above line creates the same server, the only difference is if the protocol is 'https' it will run on HTTPS server (this require SSL certificate) whereas if the protocol is http it will run on HTTP server.
Does aws-sdk for node js manage it's connections through an internal pool?
Their documentation kind of leads me to believe that.
httpOptions (map) — A set of options to pass to the low-level HTTP
request. Currently supported options are:
proxy [String] — the URL to proxy requests through agent [http.Agent,
https.Agent] — the Agent object to perform HTTP requests with. Used
for connection pooling. Defaults to the global agent
(http.globalAgent) for non-SSL connections. Note that for SSL
connections, a special Agent object is used in order to enable peer
certificate verification. This feature is only available in the
Node.js environment.
But there's no way, at least none that I could find, that'd let me define any connection pool properties.
What are my options if I want to control the concurrent connections in use?
Is it better to let the SDK handle that?
can give the http.Agent with whatever settings you want for max sockets.
var AWS = require('aws-sdk');
var http = require('http');
AWS.config.update({
httpOptions: {
agent: new http.Agent(...)
}
})
I have been looking into this a little bit more.
I dug around and figured out the defaults being used.
AWS-SDK is using the node http module, of which the defaultSocketCount is INFINITY.
They are using https module under the wraps with a maxSocketCount of 50.
The relevant code snippet.
sslAgent: function sslAgent() {
var https = require('https');
if (!AWS.NodeHttpClient.sslAgent) {
AWS.NodeHttpClient.sslAgent = new https.Agent({rejectUnauthorized: true});
AWS.NodeHttpClient.sslAgent.setMaxListeners(0);
// delegate maxSockets to globalAgent, set a default limit of 50 if current value is Infinity.
// Users can bypass this default by supplying their own Agent as part of SDK configuration.
Object.defineProperty(AWS.NodeHttpClient.sslAgent, 'maxSockets', {
enumerable: true,
get: function() {
var defaultMaxSockets = 50;
var globalAgent = https.globalAgent;
if (globalAgent && globalAgent.maxSockets !== Infinity && typeof globalAgent.maxSockets === 'number') {
return globalAgent.maxSockets;
}
return defaultMaxSockets;
}
});
}
return AWS.NodeHttpClient.sslAgent;
}
For manipulating the socket counts, see BretzL's answer.
There is however now way to set the agent for both http and https at once. You can work around this by updating the configuration as you switch from http to https and vice versa.
See : https://github.com/aws/aws-sdk-js/issues/1185
I'm creating a nodeJS server using HTTPS, similar to this:
var https = require('https');
function listener(req, res) {
// for example, I wish this worked...
console.log(req.chosen_cipher)
}
var httpsd = https.createServer(SslOptions, listener);
httpsd.listen(8081, opts.ip);
How can I find the SSL negotiation results (in particular the selected cipher), for example ECDHE-RSA-AES128-GCM-SHA256, etc.
I've serialized the req & res objects, but there doesn't seem to be any likely candidates.
Thanks!
Socket for current connection is req.client.
So, to get cipher and protocol call tlsSocket.getCipher():
function listener(req, res) {
console.log(req.client.getCipher());
// Possible result:
// { name: 'ECDHE-RSA-AES128-GCM-SHA256', version: 'TLSv1/SSLv3' }
}
I want to send a GET request using request module. Here's the code:
var requestModule = require('request');
var url = require('url');
var myUrl = 'www.google.com';
var myUrlObj = url.parse(myUrl);
requestModule(myUrl, myUrlObj , callback);
but it doesn't work because myUrlObj has a null value for its "protocol" attribute.
The same code works when:
var myUrl = 'http://www.google.com'
Why is it so rigid?
Also I tried doing the following to get around this problem:
if ( myUrlObj.protocol == null ) {
myUrl = "http://" + myUrl;
myUrlObj = url.parse(myUrl);
}
But some websites use https, while others use http. So, the above code fails for websites that use https, and the require module throws an exception.
If the URL comes from user input, default to http:// and let them enter a protocol for HTTPS. Encourage them to enter a protocol. Most HTTPS websites will redirect you from the HTTP url to the HTTPS URL. You can make the request module follow redirects using the example here.