I am deploying a Kops Kubernetes cluster on AWS with Debian Jessie image.
Mine is a hybrid environment where my artifactory is in a physical env in our DC. Now I have been facing an issue, my worker nodes are unable to pull images from my artifactory unless I specify the artifactory FQDN and IP in the /etc/hosts file.
So this is a manual edit, it works all fine after I do this. So I went ahead and added the data in my additional userdata of the Kops worker node group, but I am seeing after some time the hosts file on worker nodes is getting overwritten and also this is evident upon node reboot.
So how can I resolve this!!
The real answer is to run your own DNS server, or at least use DNS hostnames to resolve. If your router supports it, you can set local hostnames (machine-1.local)
If that isn't possible, you could try a solution like puppet if you own the virtual machines. Also, I believe Kubernetes does have a DNS addon. Also, you could use a crontab for on boot to write to the hosts file, but that's a dirty solution.
In addition, your hosts file would get rewritten for every DHCP renew. You could use static IPs, but again, DNS is the way to go.
Another workaround for this is to put it in your /etc/rc.local file:
If the file exists add this to the end:
echo '<ip-address-of-artifactory> <fqdn-of-artifactory>' >> /etc/hosts
If the file doesn't exist, create it:
$ cat << EOF > /etc/rc.local
#!/bin/sh -e
#
echo '<ip-address-of-artifactory> <fqdn-of-artifactory>' >> /etc/hosts
EOF
$ chmod 755 /etc/rc.local
$ reboot # check that it works
Related
I referred many solutions yet no luck. I have a linux automation which runs few gcloud commands with some conditions. I made this script with node js, but it is incredibly slow that I even finish it manually before the scrips completes the run.
Same with the gcloud commands when I connect to a cluster and kubectl commands when i query something.
Please help!!
It could be a DNS config error on WSL side. I hadthe same issue today, here's how I fixed it !
1. Checking the (deadly slow) response time
[tbg#~] time kubectl get deployments
No resources found in default namespace.
real 0m1.212s
user 0m0.151s
sys 0m0.050s
2. Checking the WSL/DNS configuration
[tbg#~] cat /etc/wsl.conf
[network]
generateResolvConf=false
[tbg#~] cat /etc/resolv.conf
nameserver XX.XXX.XXX.X
nameserver YYY.YY.YY.YY
nameserver 1.1.1.1
If you see that, remove these lines to get back to automatic resolv.conf generation and restart WSL (wsl --shutdown)
3. Checking the (fixed !) response time
[tbg#~] time kubectl get deployments
No resources found in default namespace.
real 0m10.530s
user 0m0.087s
sys 0m0.043s
I found out my resolv.conf configuration was causing that latency, by trying to reinstall kubectl with apt, and finding apt really slow too
Right now access to /mnt folders in WSL2 is too slow and by default at launch the entire Windows PATH is added to the Linux $PATH so any Linux binary that scans $PATH will make things unbearably slow.
To disable this feature, edit the /etc/wsl.conf to add the following section:
[interop]
appendWindowsPath = false
Avoid adding Windows Path to Linux $PATH and best for now is adding folders to the $PATH manually.
Terminate the WSL distro (wsl.exe --terminate <distro_name>) to make it immediately effective or wsl.exe --shutdown and start the terminal again.
Refer to the stack link for more information.
This seems to be a new issue with network-manager-openconect-gnome in Ubuntu 18.04+
I install sudo apt install network-manager-openconnect-gnome to get gnome integration with opeconnect and Cisco AnyConnect Compatible VPN (openconnect)
As an aside (which may actually be relevant) I do this to get *.local addresses to resolve:
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf as per systemd docs
Move dns before mdns4_minimal in /etc/nsswitch.conf
If I connect to the VPN with openconnect through the gnome network manager, VPN addresses (sites for work) do not resolve. Regular sites continue to work as expected.
If I connect to the VPN with openconnect on the command line with sudo openconnect vpn.mycompany.com, VPN addresses (sites for work) do resolve. Regular sites continue to work as expected.
I thought I would check to see if there were any differences between /etc/resolv.conf with each of these VPN connection methods and sure enough, there is one:
openconnect on the command line (working):
##VPNC_GENERATED# -- this file is generated by vpnc
# and will be overwritten by vpnc
# as long as the above mark is intact
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 10.10.10.10
nameserver 10.10.10.11
search broadband mycompany.com
openconnect gnome integration (not working):
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 192.168.1.1
nameserver 10.10.10.10
nameserver 10.10.10.11
search broadband mycompany.com
If I remove (or comment out) the nameserver 192.168.1.1, which is the difference in content between the working and not working files... everything works as expected. I can resolve addresses within the company and regular sites work as expected.
This does not happen with Fedora. Everything works out of the box. I'm not sure why the network-manager-openconnect-gnome package works differently or if there's a way I can make it work without either
Editing the file by hand.
Using the openconnect tool from the command line and keeping a terminal open running that command.
I set up a Debian 10 server to host my containers running on Docker version 19.03.2.
It currently hosts 3 DNS containers (pi-hole => bind9 => dnscrypt-proxy) which means my Debian 10 server acts as a DNS server for my LAN.
I want to add a new container. However, I can't build it because it fails when it comes to RUN apt-get update. I checked the content of the /etc/resolv.conf of the container, and the content seems right (nameserver 1.1.1.1 and nameserver 9.9.9.9, that matches with what I wrote in /etc/docker/daemon.json).
If I'm correct, the build step uses - by default - the DNS of the host, except if you specify DNS servers in /etc/default/docker or /etc/docker/daemon.json.
If the DNS servers in /etc/resolv.conf seem correct, and if the container has an Internet access (I tried a RUN ping 8.8.8.8 -c1 and it works), the build should succeed ?
I tried several things, like overwriting the content of /etc/resolv.conf with other DNS, I also rebooted the server, restarted Docker, pruned downloaded images, used the --no-cache option... I also reinstalled Docker. Nothing seems to work.
It must be somehow related to my DNS containers I guess.
Below is the content of the /etc/resolv.conf of the host (the first one is itself, as it redirects to Pi-hole).
Have you any lead to solve this issue ?
I can provide the docker-compose file of my DNS containers and the Dockerfile of my new container if you need them.
Thanking you in advance,
I have found this fix :
RUN chmod o+r /etc/resolv.conf && apt-get [....]
It works when I change the permissions.
I do not really understand why it behaves like this, if you have any lead I would be glad to know more !
I'm new to puppet, but picking it up quickly. Today, I'm running into an issue when trying to run the following:
$ puppet agent --no-daemonize --verbose --onetime
**err: Could not request certificate: getaddrinfo: Name or service not known
Exiting; failed to retrieve certificate and waitforcert is disabled**
It would appear the agent doesn't know what server to connect to. I could just specify --server on the command line, but that will be of no use to me when this runs as a daemon in production, so instead, I specify the server name in /etc/puppet/puppet.conf like so:
[main]
server = puppet.<my domain>
I do have a DNS entry for puppet.<my domain> and if I dig puppet.<my domain>, I see that the name resolves correctly.
All puppet documentation I have read states that the agent tries to connect to a puppet master at puppet by default and your options are host file trickery or do the right thing, create a CNAME in DNS, and edit the puppet.conf accordingly, which I have done.
So what am I missing? Any help is greatly appreciated!
D'oh! Need to sudo to do this! Then everything works.
I had to use the --server flag:
sudo puppet agent --server=puppet.example.org
I actually had the same error but I was using the two learning puppet vm and trying run the 'puppet agent --test' command.
I solved the problem by opening the file /etc/hosts on both the master and the agent vm and the line
***.***.***.*** learn.localdomain learn puppet.localdomain puppet
The ip address (the asterisks) was originally some random number. I had to change this number on both vm so that it was the ip address of the master node.
So I guess for experienced users my advice is to check the /etc/hosts file to make sure that the ip addresses in here for the master and agent not only match but are the same as the ip address of the master.
for other noobs like me my advice is to read the documentation more clearly. This was a step in the 'setting up an agent vm' process the I totally missed xD
In my case I was getting same error but it was due to the cert which should been signed to node on puppetmaster server.
to check pending certs run following:
puppet cert list
"node.domain.com" (SHA256) 8D:E5:8A:2*******"
sign the cert to node:
puppet cert sign node.domain.com
Had the same issue today on puppet 2.6 on CentOS 6.4
All I did to resolve the issue was to check the usual stuff such as hosts and resolv.conf to ensure they were as expected (compared with a working server) and then;
Removed /var/lib/puppet directory rm -rf /var/lib/puppet
Cleared the certificate on the puppet master puppetca --clean
servername
Restarted the network service network restart
Re-ran puppet
Even though the resolv.conf was identical to the working server, puppet updated resolv.conf and immediately re-signed the certificate and replaced all the puppet lib files.
Everything was fine after that.
I'm using an Ubuntu 8.04 (x86_64) machine to connect to my employer's Cisco VPN. (The client didn't compile out of the box, but I found patches to update the client to compile on kernels released in the last two years.) This all works great, until my DHCP client decides to renew its lease and updates /etc/resolv.conf, replacing the VPN-specific name servers with my general network servers.
Is there a good way to prevent my DHCP client from updating /etc/resolv.conf while my VPN is active?
If you are running without NetworkManager handling the connections, use the resolvconf package to act as an intermediary to programs tweaking /etc/resolv.conf: sudo apt-get install resolvconf
If you are using NetworkManager it will handle this for you, so get rid of the resolvconf package: sudo apt-get remove resolvconf
I found out about this when setting up vpnc on Ubuntu last week. A search for vpn resolv.conf on ubuntuforums.org has 250 results, many of which are very related!
If you are using the Ubuntu default with NetworkManager, try removing the CiscoVPN client and use the NetworkManager vpnc plugin to connect to the Cisco VPN. This should avoid all problems, since NetworkManager then knows about your VPN connection.
I would advice following the advice from #Sean, but if that fails for whatever reason, it should be possible to configure dhclient to not request DNS servers in /etc/dhcp3/dhclient.conf
chattr +i /etc/resolv.conf should work. ( -i to undo )
But the better thing is to configure your dhclient.conf:
https://calomel.org/dhclient.html
Look at superceding domain-name-servers, and domain-name.
Also look at "send hostname;"
If it works at your work place, you will have a cool hostname for your PC and not some weird name that DHCP servers assign.
vpnc seems to be doing the right thing for my employer's cisco concentrator. I jump on and off the vpn, and it seems to update everything smoothly.
The DHCPclient daemon can be told not to update resolv.conf with a command line switch. (-r I think, depending on the client)
That's less dynamic, because you'd have to restart/reconfigure DHCP when you connect, but not too hard. Similarly, you could just stop the service, but you might lose your IP in the meantime, so I wouldn't really recommend that.
Alternatively, you could run the dhcpclient from within a cron job, adding the appropriate process checks.
This problem is much more noticeable on networks with low DHCP lease ages. There is a bug filed in Ubuntu's dhcp3 package launchpad:
https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/90681
Which includes this patch in the description:
--- /sbin/dhclient-script.orig 2007-03-08 19:19:56.000000000 +0000
+++ /sbin/dhclient-script 2007-03-08 19:19:46.000000000 +0000
## -13,6 +13,10 ##
# The alias handling in here probably still sucks. -mdz
make_resolv_conf() {
+ # don't overwrite resolv.conf at RENEW time, since a VPN/PPTP tunnel may
+ # have updated it with remote DNS servers
+ [ "$reason" = "RENEW" ] && return
+
if [ -n "$new_domain_name" -o -n "$new_domain_name_servers" ]; then
# Find out whether we are going to mount / rw
exec 9>&0 </etc/fstab
This change to /sbin/dhcp-script stops DHCP client from overwriting /etc/resolv.conf when it renews its lease.