2

I am testing illumos in some of its variants, currently OmniOS.

As I was benchmarking io-bound processes, I saw that rsync was significantly slower in respect to my reference, FreeBSD 12-CURRENT.

Using same hardware, same command with same source and target disks:

In OmniOS r151026 I measured,

test@omniosce:~# time rsync -aPt /zarc/images /home/test/

real    17m25.428s
user    28m33.792s
sys     2m46.217s

In FreeBSD 12-CURRENT:

test@freebsd:~ % time rsync -aPt /zarc/images /home/test/

374.651u 464.028s 11:30.63 121.4%   567+210k 791583+780083io 2pf+0w

(Note that FreeBSD 12-CURRENT contains debug switches, so it runs slower than future upcoming RELEASE version).

  • I noticed that, under FreeBSD, rsync was running as 3 processes, all with nice=0, two of them consistently using 50% to 70% CPU time.

  • On OmniOS, rsync was also running as 3 processes, also with nice=0, but each one never more than 3%.

Is the CPU usage the reason execution time on same hardware is so different on FreeBSD and illumos?

If so, since nice was the same on both OS, why illumos does not allow higher CPU usage?

How could one improve rsync execution time on illumos-based OS?

Thank you in advance.

2018-06-02 edit:

  • Clarified question to make it more specific. Thanks to @rui-f-ribeiro

  • Answering to @roaima:

    1. The source and destination filesystems are both local disks
    2. This is not a one-off run for each OS, I have been testing this puzzling situation with many repetitions
    3. At ever test I am making sure the destination directory tree is completely empty of files matching those in the source
gsl
  • 298
  • 4
  • 11
  • It might depend on the quality/optimisations of the drivers, for starters...the question is interesting but risks being too broad. – Rui F Ribeiro Jun 02 '18 at 09:36
  • Thank you for comment, it is not my intention to make broad question. Could you advice how to make it more specific? Which details should I include? I am keen to learn about the reasons, and how to improve performance. – gsl Jun 02 '18 at 09:41
  • Hunt down forums about your particular hardware and development of drivers for it....you might be lucky, or not. The differences you are seeing can have several explanations, only probably doing more tests. I just remember now FreeBSD seems more willing to use native binary blobs when they exist, the difference can be explained in some cases between using a proprietary driver vs an open source one. FreeBSD also often has a large user base and developers for optimizing some corner cases. – Rui F Ribeiro Jun 02 '18 at 09:44
  • get this https://www.amazon.com/Design-Implementation-FreeBSD-Operating-System/dp/0321968972 – Rui F Ribeiro Jun 02 '18 at 09:49
  • If you are not aware, FreeBSD CURRENT is an unfinished version and there is no RELEASE version for version 12 yet. – Rob Jun 02 '18 at 11:30
  • 1
    1. Are the source and destination filesystems both local disks, or is at least one some sort of network device? 2. Is this a one-off run for each OS or is it repeatable? 3. In every case are you sure that the destination directory tree was empty of files matching those in the source? – roaima Jun 02 '18 at 14:57
  • Thank you for asking those, I have added reply to question. The issue could be OmniOS only assign small CPU usage to rsync processes? But I am not sure how to affect that. Renice does no change much. – gsl Jun 02 '18 at 15:16

1 Answers1

3

The mystery has finally been solved, by switching OmniOS rsync to 64-bit and enable optimisation. That more than doubles the speed of the checksumming code.

Now, considering exact same hardware, rysnc on OmniOS is even a bit faster than FreeBSD's.

Thank you to all here, and thanks to the excellent OmniOS devs that helped so promptly and exhaustively.

gsl
  • 298
  • 4
  • 11