10

I am tring to create random 1G test file via dd command.

dd status=progress if=/dev/zero of=/tmp/testfile.zer bs=100M count=10
dd status=progress if=/dev/urandom of=/tmp/testfile1.ran bs=100M count=10
dd status=progress if=/dev/urandom of=/tmp/testfile2.ran bs=100M count=20

The output is:

-rw-rw-r-- 1 dorinand dorinand  320M dub 21 12:37 testfile1.ran
-rw-rw-r-- 1 dorinand dorinand  640M dub 21 12:37 testfile2.ran
-rw-rw-r-- 1 dorinand dorinand 1000M dub 21 12:37 testfile.zer

Why is the output testfile generate from /dev/urandom three times smaller? I would expect that the size of testfile1.ran will be 1000M and size of testfile2.ran will be 2000M. Could anybody why this happening? How should I generate random testfile?

dorinand
  • 708
  • 1
  • 7
  • 19
  • 3
    That's because despite a widespread myth, [`dd` does not reliably copy data from its input to its output](https://unix.stackexchange.com/questions/17295/when-is-dd-suitable-for-copying-data-or-when-are-read-and-write-partial). Your question is basically a duplicate of [ Why does dd from /dev/random give different file sizes?](https://unix.stackexchange.com/questions/32988/why-does-dd-from-dev-random-give-different-file-sizes), with the twist that in the past you could get away with `dd` on Linux's `/dev/urandom`, but it seems this is no longer true. – Gilles 'SO- stop being evil' Apr 21 '18 at 20:52

1 Answers1

19

With larger blocksize, there is a risk of getting incomplete reads. This also happens a lot when reading from a pipe, rather than a block device.

If you expect to receive a certain size (count*bs) you also have to supply iflag=fullblock.

It might not be necessary for bs=1M or smaller, but it's still recommended either way.

dd will also try to show you how many incomplete reads it got. It copies n+m blocks, n complete and m incomplete ones. When copying files that are not multiple of blocksize, it's normal for the last block to be incomplete.

Example:

$ dd status=progress if=/dev/urandom of=/dev/null bs=100M count=20 
dd: warning: partial read (33554431 bytes); suggest iflag=fullblock
536870896 bytes (537 MB, 512 MiB) copied, 2 s, 254 MB/s
0+20 records in
0+20 records out
671088620 bytes (671 MB, 640 MiB) copied, 2.64391 s, 254 MB/s

In this case it got only incomplete reads and not a single full 100M block. Obviously /dev/urandom is unwilling to serve that much data in a single read. My version of dd even tells you to use iflag=fullbock directly.

With fullblock everything is OK:

$ dd status=progress if=/dev/urandom of=/dev/null bs=100M count=20 iflag=fullblock
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 8 s, 255 MB/s 
20+0 records in
20+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 8.22914 s, 255 MB/s

It takes longer because it actually copies more than twice the amount of data.

frostschutz
  • 47,228
  • 5
  • 112
  • 159
  • Did you tweak any kernel settings to get that kind of speed from `/dev/urandom`? – kasperd Apr 21 '18 at 18:12
  • 1
    @kasperd I get the same ~250 MB/s on basic Ubuntu kernel on a few years old laptop. – jpa Apr 21 '18 at 19:13
  • @jpa I tried a bunch of different Ubuntu systems, and the fastest I got was 24.3 MB/s – kasperd Apr 21 '18 at 19:18
  • @kasperd No, but the urandom algorithm changes from time to time. It was replaced with a significantly faster algorithm not too long ago. If you are using a recent-ish 4.1x.x kernel you should see improvements. – frostschutz Apr 21 '18 at 22:17
  • @frostschutz LTS 16.04 has 4.4.0.119.12, so it sounds like I'd have to switch to 18.04 to get the higher speed. (I'll switch to 18.04 at some point but I am not in a hurry since 16.04 will remain supported for quite some time to come.) – kasperd Apr 21 '18 at 22:22
  • @kasperd Well, you can use crypto layer as another fast random data source (if it's about wiping drives - stick to urandom for crypto keys and such like). – frostschutz Apr 21 '18 at 22:26
  • @kasperd Getting a bit off topic, but I'm actually on 16.04 but with the HWE kernel that is now at 4.13.0: https://wiki.ubuntu.com/Kernel/LTSEnablementStack – jpa Apr 22 '18 at 10:49
  • 1
    @jpa Yeah, I should probably ask a separate question about upgrading HWE, as it seems to not be included in `apt-get dist-upgrade`. I guess such a question would be better suited to [ubuntu.se]. – kasperd Apr 22 '18 at 11:17