Is there a way to check what the protocol is of an external site using NodeJS.
For example, for the purposes of URL shortening, people can provide a url, if they omit http or https, I'd check which it should be and add it.
I know I can just redirect users without the protocol, but I'm just curious if there is a way to check it.
Sure can. First install request-promise and its dependency, request:
npm install request request-promise
Now we can write an async function to take a URL that might be missing its protocol and, if necessary, add it:
const rq = require('request-promise');
async function completeProtocol(url) {
if (url.match(/^https?:/)) {
// fine the way it is
return url;
}
// https is preferred
try {
await rq(`https://${url}`, { method: 'HEAD' });
// We got it, that's all we need to know
return `https://${url}`;
} catch (e) {
return `http://${url}`;
}
}
Bear in mind that making requests like this could take up resources on your server particularly if someone spams a lot of these. You can mitigate that by passing timeout: 2000 as an option when calling rq.
Also consider only requesting the home page of the site, parsing off the rest of the URL, to mitigate the risk that this will be abused in some way. The protocol should be the same for the entire site.
Related
Suppose I enter a (public) website that makes 3 XHR/fetch calls on 3 different endpoints:
https://api.example.com/path1
https://api.example.com/path2
https://api.example.com/path3
What I want to achieve is intercept the call to https://api.example.com/path2 only, redirect it to a local service (localhost:8000) and let path1 and path3 through to the original domain.
What kind of options do I have here? I have studied a lot of approaches to this issue:
DNS rewriting - this solution is not suitable as I still have to intercept path1 and path3, only redirect them to the original IPs and try to mimic the headers as much as possible - which means I would have to do a specific proxy configuration for each intercepted domain - this is unfeasible
Chrome extensions - found none to deal specifically with single endpoint intercepting
Overwriting both fetch and XmlHttpRequest after page load - still doesn't cover all scenarios, maybe some websites cache the values of fetch and XmlHttpRequest before page load (?)
Combining the chrome extension and fetch overwrite will work.
download an webextension that let you load javascript code before a given page loads, e.g. User JavaScript and CSS
Add the following script to run before your page loads, base on: Intercepting JavaScript Fetch API requests and responses
const { fetch: originalFetch } = window;
window.fetch = async (...args) => {
let [resource, config ] = args;
// request interceptor starts
resource = resource === "https://api.example.com/path2" ? "http://localhost:8000/path2" : resource
// request interceptor ends
const response = await originalFetch(resource, config);
// response interceptor here
return response;
};
I am trying to do a simple fetch of pages from my Wordpress installation in my getStaticProps method:
export async function getStaticProps(context) {
// get pages!
const response = await fetch("http://localhost:8000/wp-json/wp/v2/pages");
return {
props: {
}, // will be passed to the page component as props
}
}
When calling that I get:
FetchError: request to http://localhost:8000/wp-json/wp/v2/pages failed, reason: connect ECONNREFUSED 127.0.0.1:8000
So reading the docs a little bit, I know you aren't supposed to make API calls in getStaticProps. However in my case, this is the Wordpress API, not the NextJS API, which I think is what that documentation addresses.
I am able to access my API otherwise just fine. Its like in whatever context calls 'getStaticProps', local host does not exist?
Looking at the wordpress/nextjs example where some API requests are called right within the getStaticProps, the only difference I can see is usage of a GraphQL plugin. I'm assuming that is handling some logic I am overlooking, but I am unsure.
https://github.com/vercel/next.js/tree/canary/examples/cms-wordpress
Thank you for any help in advance!
Joe
This might be a really dumb question, but is there a way to fetch without entering the server address? I'm wondering if I can just use "/init" instead of "http://localhost:3000/init"
try{
const result = await fetch("http://localhost:3001/init",
{
method:"GET",
headers:{
"content-type":"application/json"
}
});
response = await result.json();
}
catch(e){
console.log(e);
}
Is there a way to fetch without entering the server address
No.
In node.js, node-fetch requires a fully qualified URL. There is no "default" target domain or path that it could substitute like there is inside a browser web page with the browser version of fetch().
From the node-fetch documentation:
fetch(url[, options])
url should be an absolute url, such as https://example.com/.
A path-relative URL (/file/under/root) or protocol-relative URL
(//can-be-http-or-https.com/) will result in a rejected Promise.
If the problem you're really trying to solve here is to be able to write code that will work with different hosts (run locally and in a hosting environment), then you can set some sort of configuration variable with the hostname and then construct your URL using the host name in the configuration variable.
I am trying to fetch a site: link here. If you click on the link, it shows JSON: {"error":"Socket Error"}. I am trying to fetch that website, and return the error.
However, I get a 403 Forbidden error instead. Is there a reason for this? I turned CORS off, but I don't think it did anything. Here is an example of what I have tried:
async function b(){
error = await fetch('https://matchmaker.krunker.io/seek-game?hostname=krunker.io®ion=us-ca-sv&game=SV%3A4jve9&autoChangeGame=false&validationToken=QR6beUGVKUKkzwIsKhbKXyaJaZtKmPN8Rwgykea5l5FkES04b6h1RHuBkaUMFnu%2B&dataQuery=%7B%7D', {mode:'no-cors'}).then(res=>res.json())
console.log(JSON.stringify(error))
}
b()
Why doesn't anything seem to work?
Please comment if there is anything I need to add, this is my first Stack Overflow post so I am still slightly confused by what makes a good question. Thanks for helping!!
NOTE: My environment is Node.JS (testing on Repl.it which I think uses the latest Node version).
This particular host is protected width Cloudflare anti DDoS protection. The server doesn't accept requests made by fetch, but the do accept requests from curl. God knows why.
$ curl 'https://matchmaker.krunker.io/seek-game?hostname=krunker.io®ion=us-ca-sv&game=SV%3A4jve9&autoChangeGame=false&validationToken=QR6beUGVKUKkzwIsKhbKXyaJaZtKmPN8Rwgykea5l5FkES04b6h1RHuBkaUMFnu%2B&dataQuery=%7B%7D'
// => {"error":"Socket Error"}
You can use curl in node.js with node-libcurl package.
const { curly } = require('node-libcurl')
const url = 'https://matchmaker.krunker.io/seek-game?hostname=krunker.io®ion=us-ca-sv&game=SV%3A4jve9&autoChangeGame=false&validationToken=QR6beUGVKUKkzwIsKhbKXyaJaZtKmPN8Rwgykea5l5FkES04b6h1RHuBkaUMFnu%2B&dataQuery=%7B%7D'
curly.get(url)
.then(({ statusCode, data }) => console.log(statusCode, data))
// => 400 { error: 'Socket Error' }
Works as expected :-)
You can use a proxy such as allorigins.win which is a cors proxy that can retrieve the data from a URL in the form of json. You can fetch from this URL: https://api.allorigins.win/raw?url=https://matchmaker.krunker.io/game-list?hostname=krunker.io
I'm getting an SSL error when doing an HTTP.get() call in Meteor, UNABLE_TO_VERIFY_LEAF_SIGNATURE.
The links above point to solutions involving Node parameters (for instance {rejectUnauthorized: false}), but it's unclear how to pass any of those to Meteor. I've tried HTTP.get(url, {rejectUnauthorized: false}) without luck.
It's now possible by passing npmRequestOptions to Meteor HTTP requests:
const requestOptions = {
npmRequestOptions: {
rejectUnauthorized: false
}
}
const result = HTTP.get(url, requestOptions)
I ended up creating a fork of Meteor's HTTP package, which just passes through options it doesn't know about. I think it's a sane thing to do (instead of discarding the options entirely), and I hope the Meteor team pulls the change into core.
The Atmosphere package is called http-more.
Looking at the source of the HTTP package (https://github.com/meteor/meteor/blob/devel/packages/http/httpcall_server.js#L75), I noticed that it isn't implemented using node's http class directly, but instead uses the request package and the options you can pass it (see line in above link) are limited. So I'm not sure this is currently possible.
Looking at the request package's request options (https://github.com/mikeal/request#requestoptions-callback) I wouldn't be sure how to enable the option you care about either.
BTW, if you are on the server, you can always use http(s) directly using Npm.require('https').