I am exploring consul for service discovery purpose. Following is my consul cluster setup. I am having 3 consul servers & 2 consul clients machines in my cluster on CentOS 7 machine.
consulserver-01 - 192.168.30.112
consulserver-02 - 192.168.30.113
consulserver-03 - 192.168.30.114
consulclient-01 - 192.168.30.115
consulclient-02 - 192.168.30.116
I have successfully registered 3 .NET Core services named service1, service2 and service3 in this cluster which are running on 192.168.30.116 node.Now I want to resolve these services using dns. When i run dig #192.168.30.112 -p 8600 service1.service.consul SRV then it successfully resolves it. Now I want to use the systemd-resolved service to resolve it automatically. For this I have install systemd-resolved package on my consul client i.e. 192.168.30.116 machine and enter the following entries in the /etc/systemd/resolved.conf as
DNS=192.168.30.112
Domains=~consul
as per the link https://learn.hashicorp.com/tutorials/consul/dns-forwarding. I have also done the following iptables entries on my consul client i.e. 192.168.30.116
iptables -t nat -A OUTPUT -d 192.168.30.112 -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A OUTPUT -d 192.168.30.112 -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600
Now i expect that when i write ping service1.service.consul on 192.168.30.116 it should give me proper IP address i.e. 192.168.30.116.
My response for dig #192.168.30.112 -p 8600 service1.service.consul SRV is as follows
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.3 <<>> #192.168.30.112 -p 8600 service1.service.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9837
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;service1.service.consul. IN SRV
;; ANSWER SECTION:
service1.service.consul. 0 IN SRV 1 1 5555 vmdev0090.node.dc1.consul.
;; ADDITIONAL SECTION:
vmdev0090.node.dc1.consul. 0 IN A 192.168.30.116
vmdev0090.node.dc1.consul. 0 IN TXT "consul-network-segment="
;; Query time: 0 msec
;; SERVER: 192.168.30.112#8600(192.168.30.112)
;; WHEN: Thu Jul 15 11:22:25 IST 2021
;; MSG SIZE rcvd: 149
I don't know what I am doing wrong or whether my understanding is wrong. Please help in this regard.
Related
I am trying to play with socat to simulate a DNS request on localhost.
Here is what I did
setup DNS server on localhost port 15353 and once request received on 15353, it will forword it to google's DNS server 8.8.8.8:53
socat -v -v TCP-LISTEN:15353,reuseaddr,fork UDP:8.8.8.8:53
Send DNS request on same localhost using dig, specify request port to 15353
dig +tcp example.com #localhost -p 15353
Below are the response I got
On socat server terminal
socat -v -v TCP-LISTEN:15353,reuseaddr,fork UDP:8.8.8.8:53
> 2023/01/24 12:46:13.030249 length=54 from=0 to=53
.4... ........\aexample.com.......).......\f.
.\b........> 2023/01/24 12:46:23.029623 length=54 from=0 to=53
.4.i. ........\aexample.com.......).......\f.
.\b........> 2023/01/24 12:46:49.002657 length=54 from=0 to=53
.4.k. ........\aexample.com.......).......\f.
.\b...Q...0> 2023/01/24 12:46:59.001639 length=54 from=0 to=53
.4... ........\aexample.com.......).......\f.
.\b...Q...0> 2023/01/24 12:47:09.001787 length=54 from=0 to=53
.4E.. ........\aexample.com.......).......\f.
On dig client terminal
dig +tcp example.com #localhost -p 15353
; <<>> DiG 9.16.1-Ubuntu <<>> +tcp example.com #localhost -p 15353
;; global options: +cmd
;; connection timed out; no servers could be reached
Anything wrong on my command ?
Here is the working solution- using socat on UDP
set up socat DNS server redirect on localhost(127.0.0.1):
sudo socat -v -v udp4-listen:15353,reuseaddr,fork udp4:8.8.8.8:53
use dig to ask for DNS of newsblur.com on port 15353 on localhost(127.0.0.1)
dig newsblur.com #127.0.0.1 -p 15353
; <<>> DiG 9.16.1-Ubuntu <<>> newsblur.com #127.0.0.1 -p 15353
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21179
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;newsblur.com. IN A
;; ANSWER SECTION:
newsblur.com. 220 IN A 128.242.245.221
;; Query time: 4 msec
;; SERVER: 127.0.0.1#15353(127.0.0.1)
;; WHEN: Tue Jan 24 18:16:05 CST 2023
;; MSG SIZE rcvd: 46
The previous solution didn't work , seems because I was trying to transfer TCP to UDP ? I am not very sure yet , still figuring out why
I have installed PowerDNS on 2 VPS servers:
ns1 - 10.0.0.1
ns2 - 10.0.0.2
The Problem is the records/zones are not getting synced from Master to Slave. Here are the configurations:
Master Server:
allow-axfr-ips=10.0.0.2/32
daemon=yes
disable-axfr=no
include-dir=/etc/powerdns/pdns.d
master=yes
setgid=pdns
setuid=pdns
Slave Server:
daemon=yes
disable-axfr=yes
include-dir=/etc/powerdns/pdns.d
setgid=pdns
setuid=pdns
slave=yes
slave-cycle-interval=60
Database on Slave Server
MariaDB [powerdns]> select * from supermasters;
+-------------+------------------+---------+
| ip | nameserver | account |
+-------------+------------------+---------+
| 10.0.0.1 | ns2.example.com | admin |
+-------------+------------------+---------+
1 row in set (0.000 sec)
Both servers are running on MySQL database backend. Master is serving all records as expected but Slave server is giving this:
root#vps10:~# dig example.com #localhost
; <<>> DiG 9.16.1-Ubuntu <<>> example.com #localhost
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 22750
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com. IN A
;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Feb 04 22:11:39 UTC 2022
;; MSG SIZE rcvd: 45
I have also checked the Slave server and it does not have any zones from Master. Also tried this on the master server:
root#vps06:~# pdns_control notify example.com
Added to queue
Surfed the internet for any solutions but nothing was available. Can anyone guide me or point out what is wrong with my configuration?
You'll need to enable superslave and make sure your primary sends the correct notifications (NS records, ALSO-NOTIFY metadata (https://doc.powerdns.com/authoritative/domainmetadata.html?#also-notify) etc)
General context
Docker daemon comes with an embedded DNS server. It resolves local Docker swarm and network records and forwards queries for external records to an upstream nameserver configured with --dns 1.
Docs say you can set an IP address for this upstream nameserver with --dns=[IP_ADDRESS...]. The default port used is 53.
My question
Can I configure the port used as well?
My host's /etc/docker/daemon.json shows "dns": ["10.99.0.1"],. Is there a way for me to specify something like "dns": ["10.99.0.1:53"], so that dockerd always knows to forward DNS queries to port 53?
My use case
In my case, 10.99.0.1 is the IP of a localhost bridge interface. I run a local DNS caching server on this host. So DNS queries sent to 10.99.0.1:53 work. But dockerd forwards queries originating from containers connected to user-defined bridge networks (created with docker network create) to non-standard ports it picks. See terminal output below.
Detailed terminal output and debugging info
"toogle" is a Docker container connected to a Docker network I created with docker network create. 127.0.0.11 is another loopback address. DNS queries originating from within Docker containers connected to user-defined Docker networks are destined for this IP.
Is Docker's embedded DNS server actually running?
DNS queries are routed by toogle's firewall rules this way.
$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 1 packets, 60 bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 1 packets, 60 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER_OUTPUT all -- * * 0.0.0.0/0 127.0.0.11
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER_POSTROUTING all -- * * 0.0.0.0/0 127.0.0.11
Chain DOCKER_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 DNAT tcp -- * * 0.0.0.0/0 127.0.0.11 tcp dpt:53 to:127.0.0.11:37619 <-- look at this rule
0 0 DNAT udp -- * * 0.0.0.0/0 127.0.0.11 udp dpt:53 to:127.0.0.11:58552 <-- look at this rule
Chain DOCKER_POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
0 0 SNAT tcp -- * * 127.0.0.11 0.0.0.0/0 tcp spt:37619 to::53
0 0 SNAT udp -- * * 127.0.0.11 0.0.0.0/0 udp spt:58552 to::53
Whatever's listening on those ports is accepting TCP and UDP connections
$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) nc 127.0.0.11 37619 -vz
127.0.0.11: inverse host lookup failed: Host name lookup failure
(UNKNOWN) [127.0.0.11] 37619 (?) open
$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) nc 127.0.0.11 58552 -vzu
127.0.0.11: inverse host lookup failed: Host name lookup failure
(UNKNOWN) [127.0.0.11] 58552 (?) open
But there's no DNS reply from either
$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) dig #127.0.0.11 -p 58552 accounts.google.com
; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> #127.0.0.11 -p 58552 accounts.google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
$ sudo nsenter -n -t $(docker inspect --format {{.State.Pid}} toogle) dig #127.0.0.11 -p 37619 accounts.google.com +tcp
; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> #127.0.0.11 -p 37619 accounts.google.com +tcp
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
dockerd is listening for DNS queries at that IP and port from within toogle.
$ sudo nsenter -n -p -t $(docker inspect --format {{.State.Pid}} toogle) ss -utnlp
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
udp UNCONN 0 0 127.0.0.11:58552 0.0.0.0:* users:(("dockerd",pid=10984,fd=38))
tcp LISTEN 0 128 127.0.0.11:37619 0.0.0.0:* users:(("dockerd",pid=10984,fd=40))
tcp LISTEN 0 128 *:80 *:* users:(("toogle",pid=12150,fd=3))
But dockerd is trying to forward the DNS query to 10.99.0.1, which is my docker0 bridge network interface.
$ sudo journalctl --follow -u docker
-- Logs begin at Tue 2019-11-05 18:17:27 UTC. --
Apr 22 15:43:12 my-host dockerd[10984]: time="2021-04-22T15:43:12.496979903Z" level=debug msg="[resolver] read from DNS server failed, read udp 172.20.0.127:37928->10.99.0.1:53: i/o timeout"
Apr 22 15:43:13 my-host dockerd[10984]: time="2021-04-22T15:43:13.496539033Z" level=debug msg="Name To resolve: accounts.google.com."
Apr 22 15:43:13 my-host dockerd[10984]: time="2021-04-22T15:43:13.496958664Z" level=debug msg="[resolver] query accounts.google.com. (A) from 172.20.0.127:51642, forwarding to udp:10.99.0.1"
dockerd forwards the DNS query that asks for nameserver 127.0.0.11:58552 to 10.99.0.1 but only changes the IP and not the port. So the DNS query is forwarded to 10.99.0.1:58552 and nothing is listening at that port.
$ dig #10.99.0.1 -p 58552 accounts.google.com
[NO RESPONSE]
$ nc 10.99.0.1 58552 -vz
10.99.0.1: inverse host lookup failed: Unknown host
(UNKNOWN) [10.99.0.1] 58552 (?) : Connection refused
A DNS query to 10.99.0.1:53 works as expected.
dig #10.99.0.1 -p 53 accounts.google.com
; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> #10.99.0.1 -p 53 accounts.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53674
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;accounts.google.com. IN A
;; ANSWER SECTION:
accounts.google.com. 235 IN A 142.250.1.84
;; Query time: 0 msec
;; SERVER: 10.99.0.1#53(10.99.0.1)
;; WHEN: Thu Apr 22 17:20:09 UTC 2021
;; MSG SIZE rcvd: 64
I don't think there's a way to do this. I also misread the output. Docker daemon was forwarding to port 53.
read udp 172.20.0.127:37928->10.99.0.1:53: i/o timeout
I'm using this Docker image as an example to try to setup secure DNS forwarding over TLS to CloudFlare's resolvers. I'm using CoreDNS 1.5.0 (latest) and my config is this:
# CoreDNS Configuration
.:53 {
forward . tls://1.1.1.1 tls://1.0.0.1 {
tls_servername tls.cloudflare-dns.com
policy sequential
health_check 5s
}
log
}
I'm making requests like so:
root#8ef125545369:/# dig #127.0.0.1 google.com
; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> #127.0.0.1 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 49802
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 090b0d7fadcdd8bb (echoed)
;; QUESTION SECTION:
;google.com. IN A
;; Query time: 24 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Apr 08 19:29:30 UTC 2019
;; MSG SIZE rcvd: 51
I'm not getting answers. The CoreDNS logs look like this:
missioncontrol | 2019-04-08T19:29:30.778Z [INFO] 127.0.0.1:39615 - 49802 "A IN google.com. udp 51 false 4096" NOERROR - 0 5.02365452s
missioncontrol | 2019-04-08T19:29:35.759Z [INFO] 127.0.0.1:39615 - 49802 "A IN google.com. udp 51 false 4096" NOERROR - 0 5.00549558s
It's clear that CoreDNS is getting the requests, but I can't determine why this is failing. My image is ubuntu:bionic and ca-certificates is installed. I can also use openssl s_client to connect to 1.1.1.1:443 without issues.
Is there something I'm missing to setup DNS-over-TLS forwarding from CoreDNS to CloudFlare's resolvers?
EDIT
I've tested this on my host operating system outside of a Docker container and I'm seeing the same functionality, namely that it's not working.
I tested this again by running it in Travis CI, and it worked; apparently my corporate firewall does not like DNS-over-TLS.
I was able to validate this by installing knot-dnsutils (on Ubuntu 18.04) and trying to query Cloudflare directly:
$ kdig -d #1.0.0.1 +tls-ca +tls-host=cloudflare-dns.com google.com
;; DEBUG: Querying for owner(google.com.), class(1), type(1), server(1.0.0.1), port(853), protocol(TCP)
;; DEBUG: TLS, imported 133 system certificates
;; DEBUG: TLS, received certificate hierarchy:
;; DEBUG: #1, C=US,ST=California,L=San Francisco,O=Cloudflare\, Inc.,CN=cloudflare-dns.com
;; DEBUG: SHA-256 PIN: V6zes8hHBVwUECsHf7uV5xGM7dj3uMXIS9//7qC8+jU=
;; DEBUG: #2, C=US,O=DigiCert Inc,CN=DigiCert ECC Secure Server CA
;; DEBUG: SHA-256 PIN: PZXN3lRAy+8tBKk2Ox6F7jIlnzr2Yzmwqc3JnyfXoCw=
;; DEBUG: TLS, skipping certificate PIN check
;; DEBUG: TLS, The certificate is trusted.
;; WARNING: TLS, handshake failed (Error in the pull function.)
This is what happened when querying within the corporate network. From Travis CI, I saw:
;; DEBUG: Querying for owner(google.com.), class(1), type(1), server(1.1.1.1), port(853), protocol(TCP)
;; DEBUG: TLS, imported 133 system certificates
;; DEBUG: TLS, received certificate hierarchy:
;; DEBUG: #1, C=US,ST=California,L=San Francisco,O=Cloudflare\, Inc.,CN=cloudflare-dns.com
;; DEBUG: SHA-256 PIN: V6zes8hHBVwUECsHf7uV5xGM7dj3uMXIS9//7qC8+jU=
;; DEBUG: #2, C=US,O=DigiCert Inc,CN=DigiCert ECC Secure Server CA
;; DEBUG: SHA-256 PIN: PZXN3lRAy+8tBKk2Ox6F7jIlnzr2Yzmwqc3JnyfXoCw=
;; DEBUG: TLS, skipping certificate PIN check
;; DEBUG: TLS, The certificate is trusted.
;; TLS session (TLS1.2)-(ECDHE-ECDSA-SECP256R1)-(AES-256-GCM)
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 59442
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1
;; EDNS PSEUDOSECTION:
;; Version: 0; flags: ; UDP size: 1452 B; ext-rcode: NOERROR
;; PADDING: 69 B
;; QUESTION SECTION:
;; google.com. IN A
;; ANSWER SECTION:
google.com. 156 IN A 172.217.5.14
;; Received 128 B
;; Time 2019-04-09 22:03:18 UTC
;; From 1.1.1.1#853(TCP) in 12.8 ms
Clearly the corporate firewall is blocking this access unfortunately.
I setup test cluster which contains 3 servers. Consul, dnsmask and NetworkManager are installed on all machines under CentOS 7.
I'd like to test simple round robin procedure:
Expected: ping consul.service.consul must send icmp requests to one of three servers.
Actual: ping always send requests to one IP address (10.82.5.6)
However ip order is changed in answer section of dig command:
[vagrant#localhost ~]$ dig consul.service.consul
; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.1 <<>> consul.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23466
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;consul.service.consul. IN A
;; ANSWER SECTION:
consul.service.consul. 0 IN A 10.82.5.5
consul.service.consul. 0 IN A 10.82.5.4
consul.service.consul. 0 IN A 10.82.5.6
;; Query time: 2 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Dec 13 13:40:20 UTC 2017
;; MSG SIZE rcvd: 98
If I reboot 10.82.5.6 node, dig returns 2 nodes and ping begins properly work - with round robin. But when I have my node 10.82.5.6 rebooted, only this node again responds to ping commands
according to https://www.consul.io/docs/agent/dns.html the DNS interface randomizes the returned nodes so it'll never be round robin.
there's also DNS caching https://www.consul.io/docs/guides/dns-cache.html the default TTL is 0, but you may have something different and/or results are cached somewhere else..