we have machines in cluster before one week we restart the chrony service , and we check the time on all machines
after chrony restart time was ok on all machines
after 2 days we noticed that on some machine we have time diff for more then 1 hour gap
what we see is when we run the chronyd service we see from output : Can't synchronise: no selectable sources
full details
systemctl status chronyd
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2019-11-01 19:12:36 UTC; 1 day 15h ago
Process: 36664 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
Process: 36660 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 36663 (chronyd)
CGroup: /system.slice/chronyd.service
└─36663 /usr/sbin/chronyd
Nov 02 20:50:01 server_524,sys65.com chronyd[36663]: Forward time jump detected!
Nov 02 20:50:01 server_524,sys65.com chronyd[36663]: Can't synchronise: no selectable sources
Nov 02 20:53:14 server_524,sys65.com chronyd[36663]: Selected source 193.34.2.90
Nov 02 20:53:14 server_524,sys65.com chronyd[36663]: System clock wrong by -6895.552231 seconds, adjustment started
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
we also performed the following verification's , as explain down
but we not captured the root cause why time was change any suggestion what could be the reason ?
chronyc -n tracking ; date
Reference ID : 193.34.2.90 (193.34.2.90)
Stratum : 7
Ref time (UTC) : Sun Nov 3 09:23:51 2019
System time : 2563.336425781 seconds fast of NTP time
Last offset : +0.000040391 seconds
RMS offset : 0.000760717 seconds
Frequency : 30.168 ppm slow
Residual freq : +0.016 ppm
Skew : 0.907 ppm
Root delay : 0.335443 seconds
Root dispersion : 0.427117 seconds
Update interval : 70.4 seconds
Leap status : Normal
Sun Nov 3 10:07:29 UTC 2019
we compare hwclock to time on machine
date ; hwclock
Sun Nov 3 10:40:18 UTC 2019
Sun 03 Nov 2019 10:45:39 AM UTC -0.907261 seconds
the configuration of chrony is:
more /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 193.34.2.90 iburst
# Ignore stratum in source selection.
stratumweight 0
# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/drift
# Enable kernel RTC synchronization.
rtcsync
# In first three updates step the system clock instead of slew
# if the adjustment is larger than 10 seconds.
makestep 10 3
# Allow NTP client access from local network.
#allow 192.168/16
# Listen for commands only on localhost.
bindcmdaddress 127.0.0.1
bindcmdaddress ::1
# Serve time even if not synchronized to any NTP server.
#local stratum 10
keyfile /etc/chrony.keys
# Specify the key used as password for chronyc.
commandkey 1
# Generate command key if missing.
generatecommandkey
# Disable logging of client accesses.
noclientlog
# Send a message to syslog if a clock adjustment is larger than 0.5 seconds.
logchange 0.5
logdir /var/log/chrony
log measurements statistics tracking
more sanity:
1)
chronyc -n sources
210 Number of sources = 1
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* 193.34.2.90 6 6 377 15 +6522ns[ +94us] +/- 603ms
2)
timedatectl
Local time: Sun 2019-11-03 11:01:13 UTC
Universal time: Sun 2019-11-03 11:01:13 UTC
RTC time: Sun 2019-11-03 11:08:26
Time zone: Etc/UTC (UTC, +0000)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: yes
DST active: n/a
Warning: The system is configured to read the RTC time in the local time zone.
This mode can not be fully supported. It will create various problems
with time zone changes and daylight saving time adjustments. The RTC
time is never updated, it relies on external facilities to maintain it.
If at all possible, use RTC in UTC by calling
'timedatectl set-local-rtc 0'.
3)
tail -10 measurements.log
2019-11-03 09:20:19 193.34.2.90 N 6 111 111 1111 6 6 0.00 -8.100e-05 1.460e-04 1.477e-06 3.353e-01 4.268e-01
2019-11-03 09:21:30 193.34.2.90 N 6 111 111 1111 6 6 0.00 -7.400e-05 1.660e-04 1.477e-06 3.353e-01 4.279e-01
2019-11-03 09:22:40 193.34.2.90 N 6 111 111 1111 6 6 0.00 -9.400e-05 1.530e-04 1.477e-06 3.353e-01 4.290e-01
2019-11-03 09:23:51 193.34.2.90 N 6 111 111 1111 6 6 0.00 -3.800e-05 1.810e-04 1.478e-06 3.353e-01 4.300e-01
2019-11-03 09:25:01 193.34.2.90 N 6 111 111 1111 6 6 0.00 -1.900e-05 2.210e-04 1.477e-06 3.353e-01 4.311e-01
2019-11-03 09:26:11 193.34.2.90 N 6 111 111 1111 6 6 0.00 -9.500e-05 1.540e-04 1.477e-06 3.353e-01 4.321e-01
2019-11-03 09:27:21 193.34.2.90 N 6 111 111 1111 6 6 0.00 2.880e-04 9.430e-04 1.480e-06 3.353e-01 4.332e-01
2019-11-03 09:28:33 193.34.2.90 N 6 111 111 1111 6 6 0.00 -1.420e-04 1.500e-04 1.477e-06 3.353e-01 4.342e-01
2019-11-03 09:29:43 193.34.2.90 N 6 111 111 1111 6 6 0.00 -9.400e-05 1.340e-04 1.479e-06 3.353e-01 4.353e-01
2019-11-03 09:30:53 193.34.2.90 N 6 111 111 1111 6 6 0.00 -1.330e-04 2.190e-04 1.479e-06 3.353e-01 4.364e-01
4)
# tail -10 statistics.log
2019-11-03 09:25:01 193.34.2.90 5.007e-06 4.996e-06 5.989e-06 3.351e-08 3.524e-07 2.2e-02 5 0 5
2019-11-03 09:26:11 193.34.2.90 2.132e-05 8.896e-05 1.616e-05 4.535e-07 3.464e-06 1.3e+00 6 2 3
2019-11-03 09:27:21 193.34.2.90 4.463e-05 6.743e-06 4.529e-05 3.689e-07 2.964e-06 2.0e-02 5 0 4
2019-11-03 09:28:33 193.34.2.90 5.257e-05 1.377e-04 3.979e-05 9.986e-07 9.823e-06 2.3e-01 6 2 4
2019-11-03 09:29:43 193.34.2.90 4.778e-05 8.748e-05 3.340e-05 1.079e-06 8.638e-06 1.1e-02 5 1 3
==============================================================================================================
Date (UTC) Time IP Address Std dev'n Est offset Offset sd Diff freq Est skew Stress Ns Bs Nr
==============================================================================================================
2019-11-03 09:30:53 193.34.2.90 4.487e-05 1.285e-04 4.349e-05 1.598e-06 1.385e-05 6.8e-02 5 1 4
2019-11-03 09:32:03 193.34.2.90 3.720e-05 8.503e-05 3.075e-05 1.381e-06 2.748e-06 8.3e-03 5 0 5
5)
# tail -10 tracking.log
2019-11-03 09:25:01 193.34.2.90 7 -30.136 0.386 4.996e-06 N 1 5.989e-06 -2.562e+03
2019-11-03 09:26:11 193.34.2.90 7 -30.119 0.581 8.896e-05 N 1 1.616e-05 -2.556e+03
2019-11-03 09:27:21 193.34.2.90 7 -30.081 0.940 6.743e-06 N 1 4.529e-05 -2.550e+03
2019-11-03 09:28:33 193.34.2.90 7 -30.055 1.338 1.377e-04 N 1 3.979e-05 -2.544e+03
2019-11-03 09:29:43 193.34.2.90 7 -29.982 2.099 8.748e-05 N 1 3.340e-05 -2.539e+03
==================================================================================================
Date (UTC) Time IP Address St Freq ppm Skew ppm Offset L Co Offset sd Rem. corr.
==================================================================================================
2019-11-03 09:30:53 193.34.2.90 7 -29.879 3.249 1.285e-04 N 1 4.349e-05 -2.533e+03
2019-11-03 09:32:03 193.34.2.90 7 -28.764 3.389 8.503e-05 N 1 3.075e-05 -2.527e+03
6)
netstat -plaunt | egrep "chronyd"
udp 0 0 127.0.0.1:323 0.0.0.0:* 36663/chronyd
udp6 0 0 ::1:323 :::* 36663/chronyd
