Why cached reads are slower than disk reads in hdparm --direct?

Question

I am trying to interpret this result of hdparm:

janus@behemoth ~ $ sudo hdparm  -Tt --direct /dev/nvme0n1

/dev/nvme0n1:
 Timing O_DIRECT cached reads:   2548 MB in  2.00 seconds = 1273.69 MB/sec
 Timing O_DIRECT disk reads: 4188 MB in  3.00 seconds = 1395.36 MB/sec

I do not understand how the cached reads can be slower than the direct disk reads. If I drop the --direct, I get what I would have expect: the disk reads are slower than the cached ones:

janus@behemoth ~ $ sudo hdparm  -Tt /dev/nvme0n1

/dev/nvme0n1:
 Timing cached reads:   22064 MB in  2.00 seconds = 11042.86 MB/sec
 Timing buffered disk reads: 2330 MB in  3.00 seconds = 776.06 MB/sec

(Although it says "buffered disk reads" now).

Can somebody explain to me what is going on?

score 5 · Accepted Answer · answered Aug 03 '17 at 16:48

Per hdparm man page:

--direct

Use the kernel´s "O_DIRECT" flag when  performing  a  -t  timing
test.   This  bypasses  the  page cache, causing the reads to go
directly from the drive into hdparm's buffers,  using  so-called
"raw"  I/O.  In many cases, this can produce results that appear
much faster than the usual page cache method,  giving  a  better
indication of raw device and driver performance.

It so explains why hdparm -t --direct may be faster than hdparm -t. It also says that --direct only applies to the -t test, not to the -T test which is not supposed to involve the disk (see below).

-T 

Perform timings of cache reads for benchmark and comparison pur‐
poses.   For  meaningful  results,  this  operation  should   be  
repeated  2-3  times  on  an otherwise inactive system (no other
active processes) with at least a couple of  megabytes  of  free
memory.   This  displays  the speed of reading directly from the 
Linux buffer cache without disk  access.   This  measurement  is  
essentially  an  indication  of the throughput of the processor,
cache, and memory of the system under test.

I guess -T works by reading the same cached part of the disk. But your --direct prevents this. So, logically, you should have the same results with -t --direct as with -T --direct.

> So, logically, you should have the same results with `-t --direct` as with `-T --direct`. This is not the case... i think you meant "same results with `-T` as with -T --direct.`" It seems to have a nonsensical effect. Where you're reading from the cache directly with some hard drive related overhead, slowing down the cache read. — Ray Foss, Nov 14 '19 at 00:13
@Ray I don't see any contradiction in what I wrote. I just deemed 1395.36 MB/sec and 1273.69 MB/sec to be roughly the same (especially on a live system whose disks are used by other processes). I still believe that `--direct` has precedence over `-T` and that `-T --direct` gives the same results as `-t --direct`. We should look at the code to be 100% sure. — xhienne, Nov 15 '19 at 16:46

score 3 · Answer 2 · answered Nov 14 '19 at 00:52

Based on my research, --direct sends any data acquired through the hard drive at least once. That means if the system cache procures data at 10GB/sec but the poor hard drive controller bogs down at 400MB/sec, 400MB/sec is what you will get.

As a refresher

-Tt --direct Does two runs:
- 1. reads the system cache, then sends it to the hard drive controller and back. This can be useful perhaps for testing SATA connections and max buffer speeds.
- 1. Reads the disk without it's buffer.
-T reads the system cache directly, tests processor and raid driver overhead
-t reads the disk with the buffer enabled

Why cached reads are slower than disk reads in hdparm --direct?

2 Answers2