For some strange reason unknown to me, my RPi appears to have been set incorrectly to UTC +65s. The output I receive is the following:
sudo ntpd -gq
ntpd: time set -65.706156s
I have tried stopping and restarting ntp server (no effect).
When I check the sync servers using the following command, I do receive a ping back so it's not a case of the servers not responding, or a firewall issue:
grep -P "^server" /etc/ntp.conf
server 0.debian.pool.ntp.org iburst
server 1.debian.pool.ntp.org iburst
server 2.debian.pool.ntp.org iburst
server 3.debian.pool.ntp.org iburst
ping -c 1 0.debian.pool.ntp.org
PING 0.debian.pool.ntp.org (193.1.219.116) 56(84) bytes of data.
64 bytes from tbag.heanet.ie (193.1.219.116): icmp_req=1 ttl=51 time=18.8 ms
--- 0.debian.pool.ntp.org ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 18.818/18.818/18.818/0.000 ms
I'm at a loss as to how to correct this.
UPDATE:
Running the ntpq -p command yields the following info:
remote refid st t when poll reach delay offset jitter
==============================================================================
*adsl-172-10-0-1 117.70.*.110 4 u 2 64 7 0.617 -0.070 0.109
Is this the ntp server that I'm trying to sync to - because that IP belongs to CHINANET (I don't know how or why).
I also tried to manually set the RPi time, after stopping ntp service, setting the time correctly and restarting the service.
What I noticed was that the time was correctly set for a good 5 seconds, before reverting back to it's 65s offset. So it appears that this is the issue.
Found the solution as described in post 6 of the link:
http://forum.openmediavault.org/index.php/Thread/13035-Raspberry-Pi-NTP-service-not-using-etc-ntp-conf/
Basically, connecting the RPi to the network, the DHCP server acts as the NTP server and creates a copy of the ntp.conf file in the location /var/lib/ntp/ntp.conf.dhcp
This file overrides the default /etc/ntp.conf file, so deleting it and then stopping the ntp service, performing a resync, and then starting the service is the only way to resolve this.
The command for resync is:
sudo ntpdate -b pool.ntp.org
The original issue was that the ntp server was syncing with a CHINANET server and causing a 65s offset, which I suspect is down to a misconfigured DCHP/NTP server on our network.
Related
Im new to linux - so im abit confused if i have to do any best practice time sync config with Azure, or not?
From https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/accurate-time?redirectedfrom=MSDN#allowing-linux-to-use-hyper-v-host-time
The above link mentions: "For Linux guests running in Hyper-V, clients are typically configured to use the NTP daemon for time synchronization against NTP servers. If the Linux distribution supports the TimeSync version 4 protocol and the Linux guest has the TimeSync integration service enabled, then it will synchronize against the host time. This could lead to inconsistent time keeping if both methods are enabled."
How can i confirm this?
How can i confirm if TimeSync service is enabled on my RHEL 8.2 VM running in Azure?
Also how can i confirm if my ntp daemaon is configured for time synchronization against NTP servers?
As part of my investigation I have run the following on the RHEL 8.2 VM (running in Azure)
My findings on this lab are that ntp is not configured directly (/etc/ntp.conf does not exist and (as recorded in earlier comments) the ntpq command is not found,.
[user#vm-aep-dev-eastu ~]$ service ntpd status
Redirecting to /bin/systemctl status ntpd.service
Unit ntpd.service could not be found
.
however "chrony" is active.Chrony appears to be synchronising the system clock with NTP servers.
systemctl status chronyd
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-07-16 08:58:39 UTC; 7h ago
Other details:
$ /sbin/lsmod | egrep -i "^hv|hyperv"
hv_utils 36864 2
hv_balloon 28672 0
hyperv_fb 20480 1
hv_netvsc 86016 0
hv_storvsc 20480 4
hid_hyperv 16384 0
hyperv_keyboard 16384 0
hv_vmbus 114688 7 hv_balloon,hv_utils,hv_netvsc,hid_hyperv,hv_storvsc,hyperv_keyboard,hyperv_fb
Thanks
From the document Time sync for Linux VMs in Azure,
On Ubuntu 19.10 and later versions, Red Hat Enterprise Linux, and
CentOS 8.x, chrony is configured to use a PTP source clock.
For more information about Red Hat and NTP, see Configure NTP.
If both chrony and VMICTimeSync sources are enabled simultaneously,
you can mark one as prefer, which sets the other source as a backup.
Because NTP services do not update the clock for large skews except
after a long period, the VMICTimeSync will recover the clock from
paused VM events far more quickly than NTP-based tools alone.
See here for more details.
Our automated script connects to third party server (using proxy), to get some files over sftp.
It usually works fine. On a random day, I got below error. What could be the root cause.?
sftp> mget TR_ACK*.txt
Bad packet length 1131376238.
ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Connection corrupted
Connection closed
Edit: FYI- Our script runs every 5min. The above issue occured at 12:05 min. The run at 12:00 min was also failed as the server was down at 12:00. The server was just up just before the time was 12:05 minutes.
Got strange stuff. Setup: VirtualBox + CentOs7 + python3.6 + scapy2.4.0
Got network with only 4-5 hosts active: gateway, CentOs in VirtualBos, PC on which VirtualBox running and something else.
Trying to do:
ans, unans = sr(IP(dst='10.10.10.1-100')/ICMP(), iface = 'enp0s3', retry=0, timeout=1)
Begin emission: ...
Received 1822 packets, got 99 answers, remaining 1 packets
ans
Results: TCP:0 UDP:0 ICMP:99 Other:0
unans
Unanswered: TCP:0 UDP:0 ICMP:1 Other:0
ans[x] - are legit ICMP Reply packets.
unans[0] - no ICMP reply from CentOs VM itself
So looks like everything is alive instead of 4-5 hosts which actually are alive
What could be the possible reason ?
You want to know the possible reason, but scapy is not giving you enough details. So use tcpdump:
$ sudo tcpdump -e -c 200 icmp
Send the probe packets while tcpdump is running, in order to view address and timing details. It is possible you are seeing lots of perfectly normal ICMPs, for example port unreachable, or network unreachable. Tcpdump will tell you exactly what went over the network interface.
I am using ISC dhclient 4.3.1 on my debian 8.3 Linux. When the $reason reported by dhclient is TIMEOUT due to valid lease from leases file cannot be reached, the dhclient-script exits with status 2. The dhclient is not sending DHCPDISCOVER msgs after this. However, when the $reason reported by dhclient is FAILED due to no valid lease available, the dhclient keeps sending DHCPDISCOVER msgs periodically. Note that the timeout and retry values in dhclient.conf are 8 second and 1 second respectively.
There can be a scenario where the DHCP server is unreachable now, because its down or something, but can be available later. In such a case with valid lease in leases file if $reason is TIMEOUT, the dhclient is not sending DHCPDISCOVER periodically.
I am using ifup/ifdown scripts to manage the network connectivity.
Found the issue myself. It was due to dhclient-script wrongly interpreting the exit status, due to which the exit status always turned out to be 0 and hence "if (!script_go (client))" in state_panic function always turned out to be true, there by putting the dhclient in BOUND state and not starting a timer to send the DHCPDISCOVER.
We are encountering clock drift issues with our MongoDB replica set running on AWS. This just seemed to start happening recently after we added additional data to the set, before then we did not really notice this issue unless the system was under heavy load. The following error is logged in the mongod.log file sporadically and the system is not under load.
To test this we have isolated a set of machines with the same dataset and not in use by our web application though the error is still occurring;
2014-12-12T13:33:51.333+0000 [rsBackgroundSync] changing sync target
because current sync target's most recent OpTime is Dec 12 13:32:42:c
which is more than 30 seconds behind member mongo1:27017 whose most
recent OpTime is 1418391230
From the above the time stamp shows that one of the mongodb replica set members is over a minute behind. The worst we have seen is 12 minutes out of sync.
This error in turn causes replication lag and we receive the notification about this from the Mongo Monitoring Service although it does correct itself.
The setup is 3 x r3.xlarge AWS Linux instances, 1 in each availability zone of the EU-West-1A region. The machines have been setup using the Mongo recommended settings with a Raid array and the cloud formation scripts provided by Mongo. The data is around 4GB in size.
We think the issue is related to the NTP sync, by default on the AWS Linux Amazon Machine Image the ntpd service is configured to go to a pool of aws ntp servers hosted on www.pool.ntp.org.
To try and rule this out we setup our own NTP server on AWS that the MongoDB servers could sync to. The issue still occurred so we changed the maxpoll and minpoll time for the ntpd service on the mongo machines to sync the time every 16 seconds from the NTP server but the error is still occurring.
We increased the MongoDB OpLog size as well to see if that would make any difference but it didn’t.
Does anyone else encounter this type of issue? Is there something we are missing?
Cheers,
Colin.
ps -ef |grep ntp;
mongodb1
ntp 5163 1 0 Dec11 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
ec2-user 15865 15839 0 09:31 pts/2 00:00:00 grep ntp
mongodb2
ntp 4834 1 0 Dec11 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
ec2-user 19056 19029 0 09:31 pts/0 00:00:00 grep ntp
mongodb3
ntp 5795 1 0 Dec11 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
ec2-user 26199 26173 0 09:31 pts/0 00:00:00 grep ntp
cat /etc/ntp.conf;
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
driftfile /var/lib/ntp/drift
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1
# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.amazon.pool.ntp.org iburst dynamic
#server 1.amazon.pool.ntp.org iburst dynamic
#server 2.amazon.pool.ntp.org iburst dynamic
#server 3.amazon.pool.ntp.org iburst dynamic
server time-server.domain.com iburst
#broadcast 192.168.1.255 autokey # broadcast server
#broadcastclient # broadcast client
#broadcast 224.0.1.1 autokey # multicast server
#multicastclient 224.0.1.1 # multicast client
#manycastserver 239.255.254.254 # manycast server
#manycastclient 239.255.254.254 autokey # manycast client
# Enable public key cryptography.
#crypto
includefile /etc/ntp/crypto/pw
# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys
# Specify the key identifiers which are trusted.
#trustedkey 4 8 42
# Specify the key identifier to use with the ntpdc utility.
#requestkey 8
# Specify the key identifier to use with the ntpq utility.
#controlkey 8
# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats
# Enable additional logging.
logconfig =clockall =peerall =sysall =syncall
# Listen only on the primary network interface.
interface listen eth0
interface ignore ipv6
ntpq -npcrv;
remote refid st t when poll reach delay offset jitter
==============================================================================
*172.31.14.137 91.*.*.* 3 u 557 1024 377 1.121 -0.264 0.161
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.6p5#1.2349-o Sat Mar 23 00:37:31 UTC 2013 (1)",
processor="x86_64", system="Linux/3.14.23-22.44.amzn1.x86_64", leap=00,
stratum=4, precision=-23, rootdelay=23.597, rootdisp=109.962,
refid=172.31.14.137,
reftime=d83a757a.175b5fa1 Tue, Dec 16 2014 9:10:18.091,
clock=d83a77a7.82431efa Tue, Dec 16 2014 9:19:35.508, peer=27361,
tc=10, mintc=3, offset=-0.264, frequency=-13.994, sys_jitter=0.000,
clk_jitter=0.358, clk_wander=0.053
After upgrading to MongoDB 3 using the WiredTiger storage engine we do not see this issue any more.