I've try yo measure DNS latency in my docker-compose/kubernetes cluster.
setInterval(() => {
console.time('dns-test');
dns.lookup('http://my-service', (_, addresses, __) => {
console.log('addresses:', addresses);
console.timeEnd('dns-test');
});
}, 5000);
But get addresses: undefined, any ideas?
...dns.lookup('http://my-service'...
The lookup function (with example usage) takes the first parameter as the host name that you want to lookup, eg. google.com. You should remove "http://" from the name you passed in.
Related
UPDATE:
So this works as expected if I add the following:
const options = {
...standardStuff,
family: 6,
}
Adding the family: 6 option means it works as expected. So I supposed my question changes to why?? From the docs it states:
IP address family to use when resolving host or hostname.
Valid values are 4 or 6. When unspecified, both IP v4 and v6 will be used.
Whixh would leave me to conclude I wouldnt need to as IPV6 is being used anyway. And why would curl etc not matter?
Have a zone lockdown rule on cloudflare for a cname we have registered with my IPV6 address added to the white list. I got this from googling whatsmyip.
I also added my companies VPN ip address with is in the ipv4 format.
When I curl this endpoint I receive the expeted 200 - however when I run a request via nodejs I receive a 403.
This is even stranger as I am able to access the endpoint via golang, insomnia, curl and I am also able to access it via nodejs when I am connected to an ipv4 network - e.g VPN or if I tether my phone to my laptop.
curl https://my-restricted-endpoint.com
# 200
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
resp, err := http.Get("https://my-restriced-endpoint.com")
if err != nil {
log.Fatal(err)
}
fmt.Println(resp.StatusCode)
// 200
}
const requestAsync = (options: RequestOptions | string | URL): Promise<number> => {
return new Promise((resolve, reject) => {
const req = request(options, (res) => {
if (typeof res.statusCode !== 'number') {
reject('no status code returned');
return;
}
resolve(res.statusCode);
});
req.on('error', (error) => {
reject(error);
});
req.on('timeout', () => {
reject('request timed out');
});
req.end();
});
};
const statusCode = await requestAsync('https://my-restricted-endpoint.com')
// returns 200 on VPN or thetherd to phone with an ipv4 ip address
// returns 403 otherwise
My knowlege of netowkring and IPV4/6 is limited to nonexistent - but feel like this is the cause with something nodejs is doing with the request.
I have also tried using axios
So if I only whitelist my ipv6 address then I need to force nodejs to resolve the hostname to ipv6 by setting {family: 6}.
If I add both my ipv6 and ipv4 to whitelist then I can leave that option alone.
It seems that golang, curl, and insomnia in which I was using implement RFC6555 'Happy Eyeballs' which means that ipv6 will be used first and only on failure would ipv4 be used. It was why these worked and nodejs did not. From what I can gather nodejs does not implement this which means that due no whitelisting my ipv4 address on cloudflare it would fail.
I know on a Mac OSX I can run this command: dns-sd -q a5b3ef18-2e66-4e24-91d2-893b93bbc1c1.local and it returns an IP address. Can I do this in Node.js? It seems like the dns module is mainly used for website -> IP, not IP -> IP (resolved) conversions. Any help is appreciated. Thanks!
Note: The imputted addresses will be mDNS, converted by Bonjour. I found the Bonjour npm package/library, but don't think it works in this case. Also, I found mdns which has the mdns.dns_sd function but I cannot seem to figure out how to use it in my case.
Thanks!
I found a Node module that does exactly what you need.
Multicast-DNS is capable of querying mDNS IPs to the standard IP format. The snippet on their README does what you need:
var mdns = require('multicast-dns')()
mdns.on('response', function(response) {
console.log('got a response packet:', response)
})
mdns.on('query', function(query) {
console.log('got a query packet:', query)
})
// lets query for an A record for 'brunhilde.local'
mdns.query({
questions:[{
name: 'brunhilde.local',
type: 'A'
}]
})
Obviously you need to replace brunhilde.local with a valid mDNS ip. I simplified the code into this:
function query(mdns_ip){
return new Promise((resolve, reject)=>{
mdns.on('response', function(response) {
if(response.rcode === 'NOERROR'){
resolve(response.answers[0].data)
mdns.destroy()
} else {
reject(response.rcode)
mdns.destroy()
}
})
mdns.query({
questions:[{
name: mdns_ip,
type: 'A'
}]
})
})
}
I am trying to query an Amazon RDS database from a Firebase Node JS cloud function. I built the query and can successfully run the code locally using firebase functions:shell. However, when I deploy the function and call it from client-side js on my site I receive errors on both the client and server side.
Client-side:
Error: internal
Origin http://localhost:5000 is not allowed by Access-Control-Allow-Origin.
Fetch API cannot load https://us-central1-*****.cloudfunctions.net/query due to access control checks.
Failed to load resource: Origin http://localhost:5000 is not allowed by Access-Control-Allow-Origin.
Server-side:
Function execution took 60004 ms, finished with status: 'timeout'
I believe the issue has two parts:
CORS
pool.query() is async
I have looked at multiple questions for a CORS solution, here and here for example, but none of the solutions have worked for me. In regards to pool.query() being async I believe I am handling it correctly however neither the result nor an error is printed to the servers logs.
Below is all the relevant code from my projects.
Client-side:
var queryRDS = firebase.functions().httpsCallable('query');
queryRDS({
query: document.getElementById("search-input").value
})
.then(function (result) {
if (result) {
console.log(result)
}
})
.catch(function (error) {
console.log(error);
});
Server-side:
const functions = require('firebase-functions');
const { Pool } = require('pg');
const pool = new Pool({
user: 'postgres',
host: '*****.*****.us-west-2.rds.amazonaws.com',
database: '*****',
password: '*****',
port: 5432
})
exports.query = functions.https.onCall((data, context) => {
// This is not my real query, I just changed it for the
// simplicity of this question
var query = "Select * FROM table"
pool.query(query)
.then(result_set => {
console.log(result_set)
return result_set
}).catch(err => {
console.log(err)
return err
})
})
I know everything works up until pool.query(), based on my logs it seems that the .then() or the .catch() are never reached and the returns never reach the client-side.
Update:
I increased the timeout of the Firebase Functions from 60s to 120s and changed my server function code by adding a return statment before pool.query():
return pool.query(query)
.then(result_set => {
console.log(result_set)
return result_set
}).catch(err => {
console.log("Failed to execute query: " + err)
return err
})
I now get an error message reading Failed to execute query: Error: connect ETIMEDOUT **.***.***.***:5432 with the IP address being my AWS RDS database. It seems this might have been the underlying problem all along, but I am not sure why the RDS is giving me a timeout.
The CORS should be automatically handled by the onCall handler. The error message about CORS is likely to be inaccurate, and a result of the function timing out, as the server side error is showing.
That being said, according to the Cloud Functions Documentation on Function's Timeout, the default timeout for Cloud Functions is of 60 seconds, which translated to the ~60000 ms on your error message, and this means that 1 minute is not enough for your function to execute such query, which makes sense if your consider that the function is accessing an external provider, which is the Amazon RDS database.
In order to fix it you will have to redeploy your function with a flag for setting the function execution timeout, as follows:
gcloud functions deploy FUNCTION_NAME --timeout=TIMEOUT
The Value of TIMEOUT could be anything until 540, which is the maximum seconds that Cloud Functions allows before timeout (9 minutes).
NOTE: This could also be mitigated by deploying your function to the closest location possible to where your Amazon RDS database is located, you can check this link on what locations are available for Cloud Functions and you can use --region=REGION on the deploy command to specify region to be deployed.
there I want to extract the domain name of incoming request using request module. I tried by looking at
request.headers,
request.headers.hostname
with no luck. Any help please?
I figured it out. Client domain is available at origin.
request.headers.origin
for ex:
if(request.headers.origin.indexOf('gowtham') > -1) {
// do something
}
Thanks!
You should use request.headers.host .
So you want the domain name of the client that is making the request?
Since Express only provides you with their IP-address, you have to reverse-lookup that IP-address to get the hostname. From there, you can extract the domain name.
In a middleware
const dns = require('dns');
...
app.use(function(req, res, next) {
dns.reverse(req.ip, function(err, hostnames) {
if (! err) {
console.log('domain name(s):', hostnames.map(function(hostname) {
return hostname.split('.').slice(-2).join('.');
}));
}
next();
});
});
However, very big disclaimer: making a DNS lookup for every request can have a severe performance impact.
Is there a way in NodeJS to bypass linux's /etc/hosts file?
For example, if I have an entry in my hosts file like:
127.0.0.1 example.com
How would I access the 'real' example.com from NodeJS?
I don't want to temporarily modify /etc/hosts to do this, as it can bring about some problems and is not a clean solution.
Thanks in advance!
I didn't think was possible at first but then I stumbled upon a way to ignore the hosts file on Linux. Additionally, I found a DNS API built into Node. Basically, by default, Node defers to the operating system to do a DNS lookup (which will read from /etc/hosts and not make a DNS query at all if it doesn't have to). However, Node also exposes a method to resolve hostnames by explicitly making DNS requests. This will give you the IP address you're looking for.
$ cat /etc/hosts
<snip>
127.0.0.1 google.com
$ ping google.com
PING google.com (127.0.0.1) 56(84) bytes of data.
...
That shows the host file is definitely working.
const dns = require('dns')
// This will use the hosts file.
dns.lookup('google.com', function (err, res) {
console.log('This computer thinks Google is located at: ' + res)
})
dns.resolve4('google.com', function (err, res) {
console.log('A real IP address of Google is: ' + res[0])
})
This outputs different values as expected:
$ node host.js
This computer thinks Google is located at: 127.0.0.1
A real IP address of Google is: 69.77.145.253
Note that I tested this using the latest Node v8.0.0. However, I looked at some older docs and the API existed since at least v0.12 so, assuming nothing significantly changed, this should work for whatever version of Node you happen to running. Also, resolving a hostname to an IP address might be only half the battle. Some websites will behave strangely (or not at all) if you try accessing them through an IP directly.
Based on #supersam654 and this: my solution (full example) with .get request (igrnore hosts with all requests):
const dns = require("dns");
const url = require("url");
const req = (urlString, cb) => {
const parsedUrl = url.parse(urlString);
const hostname = parsedUrl.hostname;
dns.resolve4(hostname, function(err, res) {
if (err) throw err;
console.log(`A real IP address of ${hostname} is: ${res[0]}`);
const newUrl = urlString.replace(`${parsedUrl.protocol}//${hostname}`, `${parsedUrl.protocol}//${res[0]}`);
https
.get(
newUrl,
{
headers: { host: hostname },
servername: hostname
},
resp => {
let data = "";
// A chunk of data
resp.on("data", chunk => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on("end", () => {
cb(data);
});
}
)
.on("error", err => {
console.log("Error request " + url + ": " + err.message);
});
});
};
// Example
req("https://google.com/", data => console.log(data));