Precision of the date command and arithmetic

Question

I take a date at nanosecond precision:

$ start=$(date '+%s.%N')

...then print it:

$ echo ${start}
1662664850.030126174

So far so good. But look what I get when I printf with some arbitrarily huge precision:

1662664850.0301261739805340766906738281250000000000000000000000000

Q1. Did the date command actually populate the start variable with that much information, or are those digits just garbage?

Here's the second part of the question. Let's say I want to do some time math. Create an "end" timestamp:

$ end=$(date '+%s.%N')
$ echo ${end}
1662665413.471669572
$ printf "%.55f\n" ${end}
1662665413.4716695720562711358070373535156250000000000000000000000

Now I get the result I expect using bc:

$ echo $(bc <<< ${end}-${start})
563.441543398

But check out what I get when I use python or perl:

$ echo $(python -c "print(${end} - ${start})")
563.441543579
$ echo $(perl -e "print(${end} - ${start})")
563.441543579102

At a certain point, the number goes off the rails:

bc       563.441543398  
python   563.441543579  
perl     563.441543579102  

Q2. The numbers are different, but not in a way that you'd expect if it was due to rounding. What gives?

System info:
Linux 3.10.0-1160.71.1.el7.x86_64 #1 SMP Wed Jun 15 08:55:08 UTC 2022

Command info:
date (GNU coreutils) 8.22
bc 1.06.95
Python 2.7.5
perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi

Answer 1

As others have said, most decimal representations of floating point numbers cannot be represented exactly in binary formats which are the ones computers use for computations.

All you see with printf %.55f is the decimal representation of the binary approximation of the original number.

Your printf (likely) builtin command converted 1662665413.471669572 to a long double binary representation using strtold() and then passed that to the printf() function (or other function in that family such as snprintf()) for formatting (with %.55f changed to %.55Lf)

How long double are represented varies from system to system, C compiler to C compiler and CPU architecture to CPU architecture. With a printf/shell compiled with GNU cc on a GNU/Linux system on amd64 hardware, long doubles have just about enough precision to hold current timestamps with nanosecond precision.

But many if not most tools and languages including most awk implementations, perl, most shells (for those that support floating point arithmetics, not bash) use doubles instead of long doubles, and doubles invariably have only 53 bits of precision, so can't have much more than 15 decimal digits of precision¹.

That's one of the reasons why most things that deal with high precision timestamps actually represent them with two integers: a number of seconds and a number of microseconds (like the struct timeval returned by the gettimeofday() system call) or nanoseconds (like the struct timespec returned by the clock_gettime() system call).

That's why GNU date has a %s and %N rather than a floating point variant of %s, why zsh has $epochtime as an array as opposed to a floating point number.

To do calculations with such high precision timestamps, if your language / tool doesn't use long doubles, or doesn't do arbitrary precision computations in decimal without using the CPU's floating point operations (such as bc), you can do them in integer.

Most shells will do integer arithmetic with 64 bit long integers, which can hold numbers up to 9223372036854775807.

So for instance, to calculate the difference between two timestamps, expressed each as a (sec, nsec) tuple, you can do (sec2 - sec1) * 100000000 + nsec2 - nsec1 and get the difference in nanoseconds.

For example, in zsh:

zmodload zsh/datetime
start=($epochtime)
uname
end=($epochtime)
print running uname took about $((
  (end[1] - start[1]) * 1_000_000_000 + end[2] - start[2] )) nanoseconds.

Here giving:

Linux
running uname took about 2621008 nanoseconds.

You might argue that having that much precision in a shell, where running any non-builtin command (including date) takes at least a few thousand nanoseconds hardly makes sense.

Things like:

zmodload zsh/datetime
start=$EPOCHREALTIME
uname
end=$EPOCHREALTIME
printf 'running uname took about %.5g seconds\n' $(( end - start ))

Where $EPOCHREALTIME has full precision as it's built by concatenating the seconds and nanoseconds (0 padded) integers with . inbetween, but where some of that precision is lost when converting to double to do the computation, are likely good enough.

Though here, like in ksh93, you'd rather do:

typeset -F SECONDS=0
uname
printf "running uname took %g seconds\n" $SECONDS

($SECONDS, which holds the time since the shell was started can be made floating point (and reset to be used as a stopwatch) in which case you get microsecond precision).

---

^{¹ for instance, with a printf that uses doubles instead of long doubles, printf %.16g <a-16-digit-number> is not guaranteed to give you the same number (try for instance with 9.999999999999919 and the printf of zsh, gawk or perl).}

Answer 2

Binary floating point math is like this. In most programming languages, it is based on the IEEE 754 standard. The crux of the problem is that numbers are represented in this format as a whole number times a power of two; rational numbers (such as 0.1, which is 1/10) whose denominator is not a power of two cannot be exactly represented.

For more detailed explanations check this answer in SO

Answer 3

Yes, some imprecision is to be expected using floats, that is an unavoidable result of using a limited set of bits, or bytes to express a real number. Or also called a number with several fractional digits.

For example, with a shorter version of the number you used:

 ➤ printf '%.50f\n' 50.030126174

50.03012617399999999862059141264580830466002225875854

A1

Q1. Did the date command actually populate the start variable with that much information, or are those digits just garbage?

A1.1. No, date did not populate the value with that much information.

A1.2. Garbage? Well, depends on who you ask. But, for me, yes they are pretty much garbage.

A2

Q2. The numbers are different, but not in a way that you'd expect if it was due to rounding. What gives?

That is entirely the result of rounding 64 bits floats (53 bits mantissa).

No more than 15 decimal digits should be considered reliable for a double float.

Solutions

You already found that bc works fine, but here are some alternatives:

[date]
$ date -ud "1/1/1970 + $end sec - $start sec " +'%H:%M:%S.%N'
00:09:23.441543398

[bc]
$ bc <<<"$end - $start"
563.441543398

[awk] (GNU) 
$ awk -M -vPREC=200 -vend="$end" -vstart="$start" 'BEGIN{printf "%.30f\n",end - start}'
563.441543398000000000000000000000

[Perl]
$ perl -Mbignum=p,-50 -e 'print '"$end"' - '"$start"', "\n"'
563.44154339800000000000000000000000000000000000000000

[python]
$ python3 -c "from mpmath import *;mp.dps=50;print('%.30s'%(mpf('$end')-mpf('$start')));"
563.44154339800000000000000000

Integer Math

But actually both start and end are not floats. Each one is an string concatenation of two integers with a dot in the middle.

We can separate each (directly in the shell) and use almost anything to do the math, even the integer shell math:

Unreliable digits

Some may argue that we can get the mathematically exact result of the best representation of the given number.

And, yes, we can calculate a lot of binary digits:

 ➤ bc <<<'scale=100; obase=2; 50.030126174/1'
110010.0000011110110110010110010101010000010101011001110010101110011\
00101110010000100101010100000110100011110000011110111100101011110011\
11111100000010000001101100100011111010100011011101100011001110110000\
01010011000100001110101110000010000100010000100100110100001010001001\
11011111101101001010100001100000001010000101011000101100110101001100

And those are 339 binary digits.

But in any case, we have to fit that into the memory space of a float (which could have several memory representations, but probably a long double).

We may choose to talk about a float representation capable of 64 binary digits (An extended float, 80 bits in an Intel FP-87) as is the most commonly used on x86 machines by the most common linux compiler gcc. Other compilers may use something else, like a 53 bits mantissa for a 64 bits double float.

Then we must cut the binary number from above down to either of this two numbers:

110010.0000011110110110010110010101010000010101011001110010101110
110010.0000011110110110010110010101010000010101011001110010101111

From both, the one which is closer to the original is the best representation.

The exact (mathematical) decimal values of both numbers are:

50.030126173999999998620591412645808304660022258758544921875000000
50.030126174000000002090038364599422493483871221542358398437500000 

The differences from the original number are:

00.000000000000000001379408587354191695339977741241455078125000000
00.000000000000000002090038364599422493483871221542358398437500000

Thus, also mathematically, the best number is the one that ends in 0.

That is why some may argue that the result could be mathematically calculated.

And while that is true, here is the problem. That is an approximation, a good approximation, the best approximation, but an approximation anyway.

And, there is no way to know before hand (before converting the real number into a binary) what is going to be the exact magnitude of the distance between the original number and the approximation value: The approximation error.

The distances are pretty much random. The error magnitude is pretty much a random number.

That is why I say that the digits after the 18 digit (for a 64 float) are unreliable.

For 53 bits (double) anything longer than 15 digits is unreliable.

$ bc <<<"scale=20;l2=l(2)/l(10); b=53 ;d=((b-1)*l2);scale=0;d/1"
15

Formula copied from 1967 D.W.Matula paper, but it is easier to find in the C standard: C11 5.2.4.2.2p11.

If the limit is 15 digits, you can see where to cut:

1662664850.030126174
1662665413.471669572
1234567890.12345
                ^----cut here!

That is why you get some imprecision in Python and Perl at that point.

Answer 4

There is an additional source of error. date returns %s and %N from different hardware clocks, and both are integers. The nanosecond value depends on whether high-resolution timers are available: even then, man -s 7 time says "microsecond accuracy is typical of modern hardware".

Whatever the resolution of those two separate parts of the time may be, when you glue them together with a . as a decimal point, and reparse that into binary, that stitched-together value has only 15 or 16 digits of precision.

As the date since the epoch is now 10 digits, the fractional part of the time now has only 5 or 6 digits of accuracy.