I set up a Debian 10 server to host my containers running on Docker version 19.03.2.
It currently hosts 3 DNS containers (pi-hole => bind9 => dnscrypt-proxy) which means my Debian 10 server acts as a DNS server for my LAN.
I want to add a new container. However, I can't build it because it fails when it comes to RUN apt-get update. I checked the content of the /etc/resolv.conf of the container, and the content seems right (nameserver 1.1.1.1 and nameserver 9.9.9.9, that matches with what I wrote in /etc/docker/daemon.json).
If I'm correct, the build step uses - by default - the DNS of the host, except if you specify DNS servers in /etc/default/docker or /etc/docker/daemon.json.
If the DNS servers in /etc/resolv.conf seem correct, and if the container has an Internet access (I tried a RUN ping 8.8.8.8 -c1 and it works), the build should succeed ?
I tried several things, like overwriting the content of /etc/resolv.conf with other DNS, I also rebooted the server, restarted Docker, pruned downloaded images, used the --no-cache option... I also reinstalled Docker. Nothing seems to work.
It must be somehow related to my DNS containers I guess.
Below is the content of the /etc/resolv.conf of the host (the first one is itself, as it redirects to Pi-hole).
Have you any lead to solve this issue ?
I can provide the docker-compose file of my DNS containers and the Dockerfile of my new container if you need them.
Thanking you in advance,
I have found this fix :
RUN chmod o+r /etc/resolv.conf && apt-get [....]
It works when I change the permissions.
I do not really understand why it behaves like this, if you have any lead I would be glad to know more !
Related
I am deploying a Kops Kubernetes cluster on AWS with Debian Jessie image.
Mine is a hybrid environment where my artifactory is in a physical env in our DC. Now I have been facing an issue, my worker nodes are unable to pull images from my artifactory unless I specify the artifactory FQDN and IP in the /etc/hosts file.
So this is a manual edit, it works all fine after I do this. So I went ahead and added the data in my additional userdata of the Kops worker node group, but I am seeing after some time the hosts file on worker nodes is getting overwritten and also this is evident upon node reboot.
So how can I resolve this!!
The real answer is to run your own DNS server, or at least use DNS hostnames to resolve. If your router supports it, you can set local hostnames (machine-1.local)
If that isn't possible, you could try a solution like puppet if you own the virtual machines. Also, I believe Kubernetes does have a DNS addon. Also, you could use a crontab for on boot to write to the hosts file, but that's a dirty solution.
In addition, your hosts file would get rewritten for every DHCP renew. You could use static IPs, but again, DNS is the way to go.
Another workaround for this is to put it in your /etc/rc.local file:
If the file exists add this to the end:
echo '<ip-address-of-artifactory> <fqdn-of-artifactory>' >> /etc/hosts
If the file doesn't exist, create it:
$ cat << EOF > /etc/rc.local
#!/bin/sh -e
#
echo '<ip-address-of-artifactory> <fqdn-of-artifactory>' >> /etc/hosts
EOF
$ chmod 755 /etc/rc.local
$ reboot # check that it works
Ive got 4 dev VMs for four projects (all VMware Player VMs w/ubuntu 15.04 host) where each is running VNC (ports 5900, 5901, 5902, 5903) respectively.
I downloaded noVNC and saved to /var/www/html (my apache2 server on same host). Based on the ReadMe I then ran on my terminal
./utils/launch.sh --vnc localhost:5900
I received a missing websockify error, so downloaded it and placed it into the util folder. I then ran the same command and it worked! The terminal told me to Navigate to a url and sure enough I could control my VM.
However -- I'm wondering how can I use noVnc to access all 4 VM's? Is there some simple way to extend the port to a range like in iptables or firewalld?
./utils/launch.sh --vnc localhost:5900-5903
Okay, Ill answer for myself here in case it helps someone in the future...
First, create a token file where each line has a nickname, ip address, and port.
I created a file named token.list where each line looks like:
localhostnickname1: localhost:5900
localhostnickname2: localhost:5901
...
Then I use my terminal to go into the websockify folder so I can see the run file. I issue it the command:
./run --web /path/to/noVNC --target-config /path/to/token.list localhost:6080
Finally, I open my web browser and go to :
http://localhost:6080/vnc_auto.html?path=?token=localhostnickname1
Where localhost1 is the nickname of my first server on the first line of token.list
This link was my reference. If you want to serve this outside of localhost -- change the parameter localhost:8060 from localhost to an IP
PROBLEM: Any build, with a Dockerfile depending on centos, ubuntu or debian fails to build.
ENVIRONMENT: I have a Mac OS X, running VMWare with a guest Ubuntu 14.04, running Docker:
mdesales#ubuntu ~ $ sudo docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070
BEHAVIOR: Using "docker build" fails to download packages. Here's an example of such Dockerfile: https://github.com/Krijger/docker-cookbooks/blob/master/jdk8-oracle/Dockerfile, https://github.com/ottenhoff/centos-java/blob/master/Dockerfile
I know that we can run a container with --dns, but this is during the build time.
CENTOS
FROM centos
RUN yum install a b c
UBUNTU
FROM ubuntu
RUN apt-get install a b c
Users have reported that it might be problems with DNS configuration, others and the configuration has the Google's DNS servers commented out.
Step 2 : RUN yum install -y curl; yum upgrade -y; yum update -y; yum clean all
---> Running in 5f11b65c87b8
Loaded plugins: fastestmirror
Couldn't resolve host 'mirrorlist.centos.org
Still the problem persisted... So, most users on #docker#Freenode mentioned that it might be a problem with the DNS configuration... So here's my Ubuntu:
$ sudo cat /etc/resolv.conf
nameserver 127.0.1.1
search localdomain
I tried changing that, same problem...
PROBLEM
Talking to some developers at #docker#freenode, the problem was clear to everyone: DNS and the environment. The build works just fine at a regular Internet connection at home.
SOLUTION:
This problem occurs in an environment that has a private DNS server, or the network blocks the Google's DNS servers. Even if the docker container can ping 8.8.8.8, the build still needs to have access to the same private DNS server behind your firewall or Data Center.
Start the Docker daemon with the --dns switch to point to your private DNS server, just like your host OS is configured. That was found by trial and error.
Details
My MAC OS X, host OS, had a different DNS configured on my /etc/resolv.conf:
mdesales#Marcello-Work ~ (mac) $ cat /etc/resolv.conf
search corp.my-private-company.net
nameserver 172.18.20.13
nameserver 172.20.100.29
My host might be dropping the packets to the Google's IP address 8.8.8.8 while building... I just took those 2 IP addresses and placed under the Ubuntu's docker daemon configuration:
mdesales#ubuntu ~ $ cat /etc/default/docker
...
...
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 172.18.20.13 --dns 172.20.100.29 --dns 8.8.8.8"
...
The build now works as expected!
$ sudo ./build.sh
Sending build context to Docker daemon 7.168 kB
Sending build context to Docker daemon
Step 0 : FROM centos
---> b157b77b1a65
Step 1 : MAINTAINER Marcello_deSales#intuit.com
---> Running in 49bc6e233e4c
---> 2a380810ffda
Removing intermediate container 49bc6e233e4c
Step 2 : RUN yum install -y curl; yum upgrade -y; yum update -y; yum clean all
---> Running in 5f11b65c87b8
Loaded plugins: fastestmirror
Determining fastest mirrors
* base: mirror.supremebytes.com
* extras: centos.mirror.ndchost.com
* updates: mirrors.tummy.com
Resolving Dependencies
--> Running transaction check
---> Package systemd.x86_64 0:208-11.el7 will be updated
---> Package systemd.x86_64 0:208-11.el7_0.2 will be an update
---> Package systemd-libs.x86_64 0:208-11.el7 will be updated
---> Package systemd-libs.x86_64 0:208-11.el7_0.2 will be an update
--> Finished Dependency Resolution
Thanks to #BrianF and others who helped in the IRC channel!
Permanent VM Solution - UPDATE JULY 2, 2015
We now have GitHub Enterprise and CoreOS Enterprise Docker Registry in the mix... So, it was important for me to add the corporate DNS servers from the HOST machine in order to get the VM also to work.
Replacing the /etc/resolv.conf from the guest OS with the Host's /etc/resolv.conf also resolved the problem! Docker 1.7.0. I just created a new VM using Ubuntu 15.04 on VMWare Fusion and I had this problem again...
/etc/resolv.conf BEFORE
~/dev/github/public/stackedit on ⭠ master ⌚ 20:31:02
$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search localdomain
/etc/resolv.conf AFTER*
~/dev/github/public/stackedit on ⭠ master ⌚ 20:56:09
$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
search corp.mycompany.net
nameserver 10.180.194.35
nameserver 10.180.194.36
nameserver 192.168.1.1
I had pretty the same problem. The provided solution didn't help in my case. But it worked as soon I updated my Dockerfile adding environment variables for the proxy in it.
ENV HTTP_PROXY http://<proxy_host>:<port>
ENV HTTPS_PROXY http://<proxy_host>:<port>
ENV http_proxy http://<proxy_host>:<port>
ENV https_proxy http://<proxy_host>:<port>
It's likely due to your local caching name server listening on 127.0.1.1 which is not accessible from within the container.
Try putting the following into your Dockerfile:
CMD "sh" "-c" "echo nameserver 8.8.8.8 > /etc/resolv.conf"
also, just adding the nameservers from the host (in my case mac osx ) to the docker-machine vm solves the problem.
For me the problem was that my ISP blocked google's DNS (8.8.8.8) which docker uses as a fallback default.
The trick here is to find out your DNS IP and tell docker to use it.
In my case (running Ubuntu 17.04), trying to get this information from /etc/resolv.conf did not work, but I used this command:
nmcli dev show | grep IP4.DNS
Then I took this IP and added in /etc/defaults/docker:
DOCKER_OPTS="--dns 192.168.50.1"
Now restart your docker daemon, and try building again.
In my case, the issue is that our company's DNS is flawed in few ways, which requires tampering the /etc/hosts, and for docker, /etc/docker/daemon.json. That's the file which was hiding the error:
{
"dns": ["10.5...", "10.5...", "10.5..."]
}
I have backed this up and replaced with
{
"dns": ["8.8.8.8", "8.8.4.4"]
}
And it started working. I am looking for a solution that would work in all cases - on our VPN which needs the custom DNS servers as well as home on a normal network.
Note that in modern Linux, /etc/hosts is generated and DNS is managed by SystemD. I am not sure how Docker handles this, but perhaps it could be enough to point it to SystemD's fake DNS at 127.0.0.53.
Create a local repo mirror - this can also be done as a docker-mirror-packages-repo
Then run "docker build --add-host "archive.ubuntu.com:repo-docker-ip" to have the build process download from your local mirror. That is not only faster but it ensures a better reproducibility of your builds.
I am using that for the testsuite of the docker-systemctl-replacement which is testing compatibility with a number of distros each with dozens of docker rebuilds.
I'm new to puppet, but picking it up quickly. Today, I'm running into an issue when trying to run the following:
$ puppet agent --no-daemonize --verbose --onetime
**err: Could not request certificate: getaddrinfo: Name or service not known
Exiting; failed to retrieve certificate and waitforcert is disabled**
It would appear the agent doesn't know what server to connect to. I could just specify --server on the command line, but that will be of no use to me when this runs as a daemon in production, so instead, I specify the server name in /etc/puppet/puppet.conf like so:
[main]
server = puppet.<my domain>
I do have a DNS entry for puppet.<my domain> and if I dig puppet.<my domain>, I see that the name resolves correctly.
All puppet documentation I have read states that the agent tries to connect to a puppet master at puppet by default and your options are host file trickery or do the right thing, create a CNAME in DNS, and edit the puppet.conf accordingly, which I have done.
So what am I missing? Any help is greatly appreciated!
D'oh! Need to sudo to do this! Then everything works.
I had to use the --server flag:
sudo puppet agent --server=puppet.example.org
I actually had the same error but I was using the two learning puppet vm and trying run the 'puppet agent --test' command.
I solved the problem by opening the file /etc/hosts on both the master and the agent vm and the line
***.***.***.*** learn.localdomain learn puppet.localdomain puppet
The ip address (the asterisks) was originally some random number. I had to change this number on both vm so that it was the ip address of the master node.
So I guess for experienced users my advice is to check the /etc/hosts file to make sure that the ip addresses in here for the master and agent not only match but are the same as the ip address of the master.
for other noobs like me my advice is to read the documentation more clearly. This was a step in the 'setting up an agent vm' process the I totally missed xD
In my case I was getting same error but it was due to the cert which should been signed to node on puppetmaster server.
to check pending certs run following:
puppet cert list
"node.domain.com" (SHA256) 8D:E5:8A:2*******"
sign the cert to node:
puppet cert sign node.domain.com
Had the same issue today on puppet 2.6 on CentOS 6.4
All I did to resolve the issue was to check the usual stuff such as hosts and resolv.conf to ensure they were as expected (compared with a working server) and then;
Removed /var/lib/puppet directory rm -rf /var/lib/puppet
Cleared the certificate on the puppet master puppetca --clean
servername
Restarted the network service network restart
Re-ran puppet
Even though the resolv.conf was identical to the working server, puppet updated resolv.conf and immediately re-signed the certificate and replaced all the puppet lib files.
Everything was fine after that.
I'm using an Ubuntu 8.04 (x86_64) machine to connect to my employer's Cisco VPN. (The client didn't compile out of the box, but I found patches to update the client to compile on kernels released in the last two years.) This all works great, until my DHCP client decides to renew its lease and updates /etc/resolv.conf, replacing the VPN-specific name servers with my general network servers.
Is there a good way to prevent my DHCP client from updating /etc/resolv.conf while my VPN is active?
If you are running without NetworkManager handling the connections, use the resolvconf package to act as an intermediary to programs tweaking /etc/resolv.conf: sudo apt-get install resolvconf
If you are using NetworkManager it will handle this for you, so get rid of the resolvconf package: sudo apt-get remove resolvconf
I found out about this when setting up vpnc on Ubuntu last week. A search for vpn resolv.conf on ubuntuforums.org has 250 results, many of which are very related!
If you are using the Ubuntu default with NetworkManager, try removing the CiscoVPN client and use the NetworkManager vpnc plugin to connect to the Cisco VPN. This should avoid all problems, since NetworkManager then knows about your VPN connection.
I would advice following the advice from #Sean, but if that fails for whatever reason, it should be possible to configure dhclient to not request DNS servers in /etc/dhcp3/dhclient.conf
chattr +i /etc/resolv.conf should work. ( -i to undo )
But the better thing is to configure your dhclient.conf:
https://calomel.org/dhclient.html
Look at superceding domain-name-servers, and domain-name.
Also look at "send hostname;"
If it works at your work place, you will have a cool hostname for your PC and not some weird name that DHCP servers assign.
vpnc seems to be doing the right thing for my employer's cisco concentrator. I jump on and off the vpn, and it seems to update everything smoothly.
The DHCPclient daemon can be told not to update resolv.conf with a command line switch. (-r I think, depending on the client)
That's less dynamic, because you'd have to restart/reconfigure DHCP when you connect, but not too hard. Similarly, you could just stop the service, but you might lose your IP in the meantime, so I wouldn't really recommend that.
Alternatively, you could run the dhcpclient from within a cron job, adding the appropriate process checks.
This problem is much more noticeable on networks with low DHCP lease ages. There is a bug filed in Ubuntu's dhcp3 package launchpad:
https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/90681
Which includes this patch in the description:
--- /sbin/dhclient-script.orig 2007-03-08 19:19:56.000000000 +0000
+++ /sbin/dhclient-script 2007-03-08 19:19:46.000000000 +0000
## -13,6 +13,10 ##
# The alias handling in here probably still sucks. -mdz
make_resolv_conf() {
+ # don't overwrite resolv.conf at RENEW time, since a VPN/PPTP tunnel may
+ # have updated it with remote DNS servers
+ [ "$reason" = "RENEW" ] && return
+
if [ -n "$new_domain_name" -o -n "$new_domain_name_servers" ]; then
# Find out whether we are going to mount / rw
exec 9>&0 </etc/fstab
This change to /sbin/dhcp-script stops DHCP client from overwriting /etc/resolv.conf when it renews its lease.