16

On several production platforms we have observed symptoms which appear to suggest that the time of day clock is periodically jumping forward or backward. The jumps are typically around 1 second, typically cancel out (jump forward then backward very shortly thereafter) and happen around 50 times per day. This drift is most noticeable during times of peak application usage, and during periods of high disk I/O operations such as daily backups. These drifts are affecting our soft real-time sensitive application.

Systems are Oracle Netra X4250 and Netra X4270 servers running SLES 11SP2 with 3.0.58-0.6.6-default kernel.

$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc hpet acpi_pm

$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

We have disabled NTP, but that has not had any effect on the drifts. Are there tools which measure time of day clock drift? How can we avoid this?

These are production platforms, and we cannot recreate the issue in our labs, so my ability to experiment is limited. If left to my own devices, I'll write a tool to measure drift, and perhaps experiment with an HPET clocksource.

Peter Mortensen
  • 1,029
  • 1
  • 8
  • 10
brett
  • 301
  • 1
  • 2
  • 6
  • 5
    Disabling NTP makes clocks *much more* unstable... the only reason I can see for NTP not to keep the clock in line is that the clock is out of whack, and NTP refuses to update it (see `ntpdate(8)` or `ntpd(8)`). – vonbrand Mar 07 '14 at 23:39
  • 1
    NTPD does track and correct for clock drift, but what you have isn't drift. Drift is consistently in the same direction by roughly the same amount over time. If it randomly jumps forward and backwards, there's no way to predict it, and accommodate for it. – phemmer Mar 08 '14 at 07:47
  • 1
    What @Patrick said is right, the problem you describe is a discontinuous jump in time forward and backward, multiple times per day. NTP works well on drift but it won't help you much with this. Something is likely resetting your system date to some external time source that maybe only has 1 second resolution. If your servers are x86* the hardware RTC might be the source and some cron job the culprit. As far as measuring clock offset Bratchley's ntpdate answer is a reasonable approach provided a good stratum 1 clock reference is used: run once a minute and gnuplot the result for a picture. – duanev Jul 28 '15 at 22:19
  • 1
    Ran across this evaluation of NTP starting up on a new server (http://www.drdobbs.com/embedded-systems/testing-and-verifying-time-synchronizati/223000197?pgno=2). It takes NTP hours to learn a new crystal. For really bad crystals NTP will have to 'step' the clock by significant amounts multiple times while training (see Figs 4 and 5 in that article). A final value in ntp.drift of 118ppm is 10 seconds per day or 208ms every 30 minutes. Although this is not what the OP was seeing, NTP can **initially** cause noticeable jumps in time. – duanev Jul 29 '15 at 16:11

4 Answers4

9

Are there tools which measure time of day clock drift?

The only tools I'm aware of are the NTP tools which should suffice. You don't have to actually configure ntpd to sync against a given clock source you can just use the -d option to ntpdate to fetch the calculated offset.

Example:

[davisja5@xxxadmvlm08 ~]$ ntpdate -d clock.redhat.com 2>/dev/null | egrep "^offset"
offset -0.004545
[davisja5@xxxadmvlm08 ~]$

-d is the debug option which does the NTP work without actually touching the system clock.

Any advice on how we can avoid this?

I'm not too surprised that you aren't able to reproduce this in dev/test environments since it's probably just due to the hardware clock. If you have hardware support with someone, I would try to get your machines serviced. One possibility is trading out one of the dev machines for this production machine, fixing the former PROD systems and re-introduce it as a dev machine to replace the one that's in PROD now.

Short of that, switching the hardware clock source is about all you can do. If you don't or can't do the swap thing I'd suggest that you do go the hpet route. You can test whether the clock source change messes with system services and then deploy it into production as a hail mary.

Dani_l
  • 4,720
  • 1
  • 18
  • 34
Bratchley
  • 16,684
  • 13
  • 64
  • 103
  • By "measure clock drift", I didn't mean drift from a reference time source, such as NTP gives you. I meant a tool which can detect "jumps" in the time of day clock over a continuous time range. For example, take time of day samplings every 50ms, and report if the difference from the last sampling is too far from 50ms. Such a tool would show if the time of day clock is drifting from the underlying hardware clock for any reason. – brett Mar 07 '14 at 21:21
  • 1
    Wouldn't the presence of such intervention likely cause more performance degradation than you're hoping to resolve? In all likelihood though, it's a hardware problem, so you're going to need to get the hardware serviced or use a clock source without this issue. `tsc` is based in the CPU so it makes sense that higher CPU activity would trigger an issue with the hardware clock anyways. If hpet is fast enough for you, then you may just have to try that, get serviced, or do the swap thing. Those are the only options I can see for you. – Bratchley Mar 07 '14 at 22:39
3

One solution is to use HPET

See also High Precision Event Timer

To set it as boot parameter use

clocksource=hpet

On older hardware the TSC was often unstable and was disabled by the kernel.

With the advent of multi-core/hyper-threaded CPUs, systems with multiple CPUs, and hibernating operating systems, the TSC cannot be relied on to provide accurate results ...

Wikipedia:Time Stamp Counter

Community
  • 1
  • On a production system exhibiting the clock jitter symptoms I switched the clocksource to hpet. This had no effect on the observed clock jitter symptoms. – brett Mar 12 '14 at 13:50
  • HPET is external hardware timer and cannot jitter. So this solution seems to be a wrong path. There were a lot of timing issues with older hardware, especially when using virtualization. Did you checked this with different software also? –  Mar 12 '14 at 14:46
1

I wrote a more detailed tool to correlate clock measurements with latency symptoms exhibited by our application. This tool seems to rule out what I previously suspected as jitter in the Linux time of day clock.

So long story short, my initial hypothesis was invalid. But I learned a lot about Linux clocks from the answers and links, so thanks to all who responded!

brett
  • 301
  • 1
  • 2
  • 6
0

Isn't clock supposed to be monotonous unless someone changes it? Backward jumps shouldn't be possible. There must be something setting the clock - a cron job or some other daemon (for instance a call to hwclock --adjust). I do recall that ntp itself updates statistics for drift and compensates for it routinely and if you fail to run ntp for a long time and get a huge offset, it messes up time for days after it if you don't reset /etc/adjtime. You may have something like that set up - something that readjusts time drift periodically (and causes jumps).

ntp is actually meant to counter this problem.

orion
  • 12,302
  • 2
  • 31
  • 41
  • That's what I thought as well. My reading of the hardware clock sources suggests that the counter should be monotonically increasing. If that were true, at worst we should observe erratic tick rates, but never jumps back. On a multiprocessor system, I understand that tsc needs to be synchronized amongst the processors - perhaps this is what is causing backwards jumps? – brett Mar 10 '14 at 16:02