I have 3 new fresh installs of Ubuntu 16.04.2 LTS xenial on Azure VM, in the system log I noticed I have a time discrepancy and the system is logging this ever 5 seconds.
Mar 5 17:57:57 server1 systemd[1]: snapd.refresh.timer: Adding 2h 17min 4.279485s random time.
Mar 5 17:57:57 server1 systemd[1]: apt-daily.timer: Adding 5h 14min 48.381690s random time.
Mar 5 17:57:57 server1 systemd[19425]: Time has been changed
Mar 5 17:57:57 server1 systemd[37054]: Time has been changed
I have stopped the two services: apt-daily.timer and snapd.refresh.timer, and the "Time has been changed" messages still persist. It seems to be a time discrepancy between the VM and host system. I am not sure how to address this. I also have VMs of the same exact version that I installed over a month ago on Azure and they don't show this error.
Thanks for guidance on this
Related
Amazon Linux 1:
Server version: Apache/2.4.54 (Amazon)
Server built: Jul 11 2022 21:47:38
Server's Module Magic Number: 20120211:124
Server loaded: APR 1.6.3, APR-UTIL 1.5.4, PCRE 8.21 2011-12-12
PHP 7.3.30 (cli) (built: Oct 6 2021 20:34:22) ( NTS )
Amazon Linux 2:
Server version: Apache/2.4.54 ()
Server built: Jun 30 2022 11:02:23
Server's Module Magic Number: 20120211:124
Server loaded: APR 1.7.0, APR-UTIL 1.6.1, PCRE 8.32 2012-11-30
PHP 7.4.30 (cli) (built: Jun 23 2022 20:19:00) ( NTS )
The server's are configured via automation and loaded into ALB/ASG's.
Instance size is m4.large (2x vCPU, 8GiB Memory)
Auto-Scaling group is configured with Min:4 Max:8
This is what my httpd.conf file looks like:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxConnectionsPerChild 4000
</IfModule>
On Amazon Linux 1, the site works as expected. The ASG spawns 4 instances and they each hover at 40-60% CPU utilization during peak hours.
The same build on Amazon Linux 2 yields wildly different results. All instances immediately get bombarded by a huge number of httpd processes.
The ASG scales up to 8 instances with every single one at 90%+ CPU usage. The instances start to lock up which causes the target group to mark them as "unhealthy" and the ASG to rotate them out endlessly. The website obviously does not work.
What could be causing them to behave so differently? What steps can I take to try and mitigate this? I'm honestly pretty new to all this so I don't know where to start.
I just installed filebeat, logstash, kibana and elasticsearch all running smoothly just to trial this product out for additional monthly reports/monitoring and noticed every time I try to change the "/etc/elasticsearch/elasticsearch.yml" config file for remote web access it'll basically crash the service every time I make the change.
Just want to say I'm new to the forum and this product, and my end goal for this question is to figure out how to allow remote connections to access elastisearch as I guinea pig and test without crashing elasticsearch.
For reference here is the error code when I run the 'sudo systemctl status elasticsearch' query:
Dec 30 07:27:37 ubuntu systemd[1]: Starting Elasticsearch...
Dec 30 07:27:52 ubuntu systemd-entrypoint[4067]: ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
Dec 30 07:27:52 ubuntu systemd-entrypoint[4067]: bootstrap check failure [1] of [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.se>
Dec 30 07:27:52 ubuntu systemd-entrypoint[4067]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/elasticsearch.log
Dec 30 07:27:53 ubuntu systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/CONFIG
Dec 30 07:27:53 ubuntu systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Dec 30 07:27:53 ubuntu systemd[1]: Failed to start Elasticsearch.
Any help on this is greatly appreciated!
I've been trying to troubleshoot this problem for some days now.
A couple of minutes after starting an SSH connection to my Namecheap server (on Mac/windows/cPanel's "Terminal"), it crashes and give the following error message :
Error: The connection to the server ended in failure at {TIME} PM. (SIGKILL)
and :
Exit Code: 137
I've tried to create some kind of log file for any SIGKILL signal, but, it seems like none can be made on a Namecheap server :
auditctl doesn't exist,
We can't get systemtap because no package managers are available.
Precision :
uname -a : Linux [-n] 2.6.32-954.3.5.lve1.4.78.el6.x86_64 #1 SMP Thu Mar 26 08:20:27 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
I calculated the time between each crash : around 6min.
I don't have a very good knowledge of Linux servers, and maybe didn't include needed information. So please ask for any specificities!
Im new to linux - so im abit confused if i have to do any best practice time sync config with Azure, or not?
From https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/accurate-time?redirectedfrom=MSDN#allowing-linux-to-use-hyper-v-host-time
The above link mentions: "For Linux guests running in Hyper-V, clients are typically configured to use the NTP daemon for time synchronization against NTP servers. If the Linux distribution supports the TimeSync version 4 protocol and the Linux guest has the TimeSync integration service enabled, then it will synchronize against the host time. This could lead to inconsistent time keeping if both methods are enabled."
How can i confirm this?
How can i confirm if TimeSync service is enabled on my RHEL 8.2 VM running in Azure?
Also how can i confirm if my ntp daemaon is configured for time synchronization against NTP servers?
As part of my investigation I have run the following on the RHEL 8.2 VM (running in Azure)
My findings on this lab are that ntp is not configured directly (/etc/ntp.conf does not exist and (as recorded in earlier comments) the ntpq command is not found,.
[user#vm-aep-dev-eastu ~]$ service ntpd status
Redirecting to /bin/systemctl status ntpd.service
Unit ntpd.service could not be found
.
however "chrony" is active.Chrony appears to be synchronising the system clock with NTP servers.
systemctl status chronyd
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-07-16 08:58:39 UTC; 7h ago
Other details:
$ /sbin/lsmod | egrep -i "^hv|hyperv"
hv_utils 36864 2
hv_balloon 28672 0
hyperv_fb 20480 1
hv_netvsc 86016 0
hv_storvsc 20480 4
hid_hyperv 16384 0
hyperv_keyboard 16384 0
hv_vmbus 114688 7 hv_balloon,hv_utils,hv_netvsc,hid_hyperv,hv_storvsc,hyperv_keyboard,hyperv_fb
Thanks
From the document Time sync for Linux VMs in Azure,
On Ubuntu 19.10 and later versions, Red Hat Enterprise Linux, and
CentOS 8.x, chrony is configured to use a PTP source clock.
For more information about Red Hat and NTP, see Configure NTP.
If both chrony and VMICTimeSync sources are enabled simultaneously,
you can mark one as prefer, which sets the other source as a backup.
Because NTP services do not update the clock for large skews except
after a long period, the VMICTimeSync will recover the clock from
paused VM events far more quickly than NTP-based tools alone.
See here for more details.
I have 3 servers, 2 on AWS and one on Digital Ocean, and the timezone for all is set to CDT. But when I check the current time on all 3 by using the date command via command line, none of them matches.
Server1: Wed Jun 12 23:36:01 CDT 2019
Server2: Wed Jun 12 23:45:51 CDT 2019
Server3: Wed Jun 12 23:38:39 CDT 2019
Could anyone please suggest what needs to be done here? Thanks.
Since you have not explicitly said that you have ntp running on them, you'll need to install that. Once that is installed and set up properly, you should show the same exact time on all of them.