Im new to linux - so im abit confused if i have to do any best practice time sync config with Azure, or not?
From https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/accurate-time?redirectedfrom=MSDN#allowing-linux-to-use-hyper-v-host-time
The above link mentions: "For Linux guests running in Hyper-V, clients are typically configured to use the NTP daemon for time synchronization against NTP servers. If the Linux distribution supports the TimeSync version 4 protocol and the Linux guest has the TimeSync integration service enabled, then it will synchronize against the host time. This could lead to inconsistent time keeping if both methods are enabled."
How can i confirm this?
How can i confirm if TimeSync service is enabled on my RHEL 8.2 VM running in Azure?
Also how can i confirm if my ntp daemaon is configured for time synchronization against NTP servers?
As part of my investigation I have run the following on the RHEL 8.2 VM (running in Azure)
My findings on this lab are that ntp is not configured directly (/etc/ntp.conf does not exist and (as recorded in earlier comments) the ntpq command is not found,.
[user#vm-aep-dev-eastu ~]$ service ntpd status
Redirecting to /bin/systemctl status ntpd.service
Unit ntpd.service could not be found
.
however "chrony" is active.Chrony appears to be synchronising the system clock with NTP servers.
systemctl status chronyd
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-07-16 08:58:39 UTC; 7h ago
Other details:
$ /sbin/lsmod | egrep -i "^hv|hyperv"
hv_utils 36864 2
hv_balloon 28672 0
hyperv_fb 20480 1
hv_netvsc 86016 0
hv_storvsc 20480 4
hid_hyperv 16384 0
hyperv_keyboard 16384 0
hv_vmbus 114688 7 hv_balloon,hv_utils,hv_netvsc,hid_hyperv,hv_storvsc,hyperv_keyboard,hyperv_fb
Thanks
From the document Time sync for Linux VMs in Azure,
On Ubuntu 19.10 and later versions, Red Hat Enterprise Linux, and
CentOS 8.x, chrony is configured to use a PTP source clock.
For more information about Red Hat and NTP, see Configure NTP.
If both chrony and VMICTimeSync sources are enabled simultaneously,
you can mark one as prefer, which sets the other source as a backup.
Because NTP services do not update the clock for large skews except
after a long period, the VMICTimeSync will recover the clock from
paused VM events far more quickly than NTP-based tools alone.
See here for more details.
Related
I have a VM backed by vCenter.
vCenter ESXi have physical adapter "Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller" and SR-IOV enabled on this.
VM is connected to 1mgmt network (vmxnet3) and 2 SR-IOV adapters (SRIOVPassthrough).
Upon booting of the VM, only 2 networks shown up. (1mgmt and 1SR-IOV).
Journalctl -k logs showed following error.
[ 4832.408471] bnxt_en 0000:13:00.0 (unnamed net_device) (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
[ 4832.408930] bnxt_en: probe of 0000:13:00.0 failed with error -1
Reboot of machine did not help at all.
For the successful one adapter
bnxt_en 0000:03:00.0 eth1: NIC Link is Up, 25000 Mbps full duplex, Flow control: ON - receive & transmit
bnxt_en 0000:03:00.0 eth1: FEC autoneg off encodings: None
I did rescan of the pci devices and did multiple times reboot without any success.
Any pointers would be really helpful
We've got a similar issue and were able to fix it.
In our case we had the same error message on Debian 10, 11 and Oracle Linux 8 but we installed it directly on hardware without an hypervisor.
But it could be the same issue cause you're using passthrough.
There are two ways to fix it:
Usage of UEFI Boot
Disable PXE Boot and keep Bios / Legacy Boot
Both options fixed it.
Disabling PXE didn't work for us, but we can get the ports back online, by running
echo 0000:af:00.0 > /sys/bus/pci/drivers/bnxt_en/bind
Where 0000:af:00.0 is the PCI number for the port, which can be gotten from dmesg | grep bnxt_en and looking for the port or ports that failed.
Where are my iptables logging Blocked messages? I wonder if this is an OpenVZ issue or something from the scripted install. Note, I'm highly technical, but not a server admin. Could the OpenVZ host be blocking and logging outside of my VSP?
I have two newly installed machines running running text-mode CentOS 7 x64, yum up to date packages, and with iptables/CSF.
Also, I ensured machine #2 has all the packages that are on machine #1, though #2 has some extras.
OpenVZ VPS (installed with their image of CentOS 7 x64)
VMware VM (installed with official CentOS 7 x64 minimal mode)
I performed my extra installs/configs exactly the same on both machines, and I have these lines in /etc/csf/csf.conf
TESTING = "0"
TCP_IN = "22,80,443"
UDP_IN = ""
On the VM, I'm getting these /var/log/messages when I nmap scan it:
Apr 12 17:25:23 mach kernel: Firewall: *UDP_IN Blocked* IN=ens192 OUT= ...
Apr 12 17:25:55 mach kernel: Firewall: *TCP_IN Blocked* IN=ens192 OUT= ...
On the VPS, I'm NOT getting any Firewall /var/log/messages when I nmap scan it... but I think it is properly blocking traffic.
How do I even proceed/diagnose this?
I have 3 new fresh installs of Ubuntu 16.04.2 LTS xenial on Azure VM, in the system log I noticed I have a time discrepancy and the system is logging this ever 5 seconds.
Mar 5 17:57:57 server1 systemd[1]: snapd.refresh.timer: Adding 2h 17min 4.279485s random time.
Mar 5 17:57:57 server1 systemd[1]: apt-daily.timer: Adding 5h 14min 48.381690s random time.
Mar 5 17:57:57 server1 systemd[19425]: Time has been changed
Mar 5 17:57:57 server1 systemd[37054]: Time has been changed
I have stopped the two services: apt-daily.timer and snapd.refresh.timer, and the "Time has been changed" messages still persist. It seems to be a time discrepancy between the VM and host system. I am not sure how to address this. I also have VMs of the same exact version that I installed over a month ago on Azure and they don't show this error.
Thanks for guidance on this
For some strange reason unknown to me, my RPi appears to have been set incorrectly to UTC +65s. The output I receive is the following:
sudo ntpd -gq
ntpd: time set -65.706156s
I have tried stopping and restarting ntp server (no effect).
When I check the sync servers using the following command, I do receive a ping back so it's not a case of the servers not responding, or a firewall issue:
grep -P "^server" /etc/ntp.conf
server 0.debian.pool.ntp.org iburst
server 1.debian.pool.ntp.org iburst
server 2.debian.pool.ntp.org iburst
server 3.debian.pool.ntp.org iburst
ping -c 1 0.debian.pool.ntp.org
PING 0.debian.pool.ntp.org (193.1.219.116) 56(84) bytes of data.
64 bytes from tbag.heanet.ie (193.1.219.116): icmp_req=1 ttl=51 time=18.8 ms
--- 0.debian.pool.ntp.org ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 18.818/18.818/18.818/0.000 ms
I'm at a loss as to how to correct this.
UPDATE:
Running the ntpq -p command yields the following info:
remote refid st t when poll reach delay offset jitter
==============================================================================
*adsl-172-10-0-1 117.70.*.110 4 u 2 64 7 0.617 -0.070 0.109
Is this the ntp server that I'm trying to sync to - because that IP belongs to CHINANET (I don't know how or why).
I also tried to manually set the RPi time, after stopping ntp service, setting the time correctly and restarting the service.
What I noticed was that the time was correctly set for a good 5 seconds, before reverting back to it's 65s offset. So it appears that this is the issue.
Found the solution as described in post 6 of the link:
http://forum.openmediavault.org/index.php/Thread/13035-Raspberry-Pi-NTP-service-not-using-etc-ntp-conf/
Basically, connecting the RPi to the network, the DHCP server acts as the NTP server and creates a copy of the ntp.conf file in the location /var/lib/ntp/ntp.conf.dhcp
This file overrides the default /etc/ntp.conf file, so deleting it and then stopping the ntp service, performing a resync, and then starting the service is the only way to resolve this.
The command for resync is:
sudo ntpdate -b pool.ntp.org
The original issue was that the ntp server was syncing with a CHINANET server and causing a 65s offset, which I suspect is down to a misconfigured DCHP/NTP server on our network.
We are encountering clock drift issues with our MongoDB replica set running on AWS. This just seemed to start happening recently after we added additional data to the set, before then we did not really notice this issue unless the system was under heavy load. The following error is logged in the mongod.log file sporadically and the system is not under load.
To test this we have isolated a set of machines with the same dataset and not in use by our web application though the error is still occurring;
2014-12-12T13:33:51.333+0000 [rsBackgroundSync] changing sync target
because current sync target's most recent OpTime is Dec 12 13:32:42:c
which is more than 30 seconds behind member mongo1:27017 whose most
recent OpTime is 1418391230
From the above the time stamp shows that one of the mongodb replica set members is over a minute behind. The worst we have seen is 12 minutes out of sync.
This error in turn causes replication lag and we receive the notification about this from the Mongo Monitoring Service although it does correct itself.
The setup is 3 x r3.xlarge AWS Linux instances, 1 in each availability zone of the EU-West-1A region. The machines have been setup using the Mongo recommended settings with a Raid array and the cloud formation scripts provided by Mongo. The data is around 4GB in size.
We think the issue is related to the NTP sync, by default on the AWS Linux Amazon Machine Image the ntpd service is configured to go to a pool of aws ntp servers hosted on www.pool.ntp.org.
To try and rule this out we setup our own NTP server on AWS that the MongoDB servers could sync to. The issue still occurred so we changed the maxpoll and minpoll time for the ntpd service on the mongo machines to sync the time every 16 seconds from the NTP server but the error is still occurring.
We increased the MongoDB OpLog size as well to see if that would make any difference but it didn’t.
Does anyone else encounter this type of issue? Is there something we are missing?
Cheers,
Colin.
ps -ef |grep ntp;
mongodb1
ntp 5163 1 0 Dec11 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
ec2-user 15865 15839 0 09:31 pts/2 00:00:00 grep ntp
mongodb2
ntp 4834 1 0 Dec11 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
ec2-user 19056 19029 0 09:31 pts/0 00:00:00 grep ntp
mongodb3
ntp 5795 1 0 Dec11 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
ec2-user 26199 26173 0 09:31 pts/0 00:00:00 grep ntp
cat /etc/ntp.conf;
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
driftfile /var/lib/ntp/drift
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1
# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.amazon.pool.ntp.org iburst dynamic
#server 1.amazon.pool.ntp.org iburst dynamic
#server 2.amazon.pool.ntp.org iburst dynamic
#server 3.amazon.pool.ntp.org iburst dynamic
server time-server.domain.com iburst
#broadcast 192.168.1.255 autokey # broadcast server
#broadcastclient # broadcast client
#broadcast 224.0.1.1 autokey # multicast server
#multicastclient 224.0.1.1 # multicast client
#manycastserver 239.255.254.254 # manycast server
#manycastclient 239.255.254.254 autokey # manycast client
# Enable public key cryptography.
#crypto
includefile /etc/ntp/crypto/pw
# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys
# Specify the key identifiers which are trusted.
#trustedkey 4 8 42
# Specify the key identifier to use with the ntpdc utility.
#requestkey 8
# Specify the key identifier to use with the ntpq utility.
#controlkey 8
# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats
# Enable additional logging.
logconfig =clockall =peerall =sysall =syncall
# Listen only on the primary network interface.
interface listen eth0
interface ignore ipv6
ntpq -npcrv;
remote refid st t when poll reach delay offset jitter
==============================================================================
*172.31.14.137 91.*.*.* 3 u 557 1024 377 1.121 -0.264 0.161
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.6p5#1.2349-o Sat Mar 23 00:37:31 UTC 2013 (1)",
processor="x86_64", system="Linux/3.14.23-22.44.amzn1.x86_64", leap=00,
stratum=4, precision=-23, rootdelay=23.597, rootdisp=109.962,
refid=172.31.14.137,
reftime=d83a757a.175b5fa1 Tue, Dec 16 2014 9:10:18.091,
clock=d83a77a7.82431efa Tue, Dec 16 2014 9:19:35.508, peer=27361,
tc=10, mintc=3, offset=-0.264, frequency=-13.994, sys_jitter=0.000,
clk_jitter=0.358, clk_wander=0.053
After upgrading to MongoDB 3 using the WiredTiger storage engine we do not see this issue any more.