2

Observing the output of the tar command (-v flag) - observation by human eye - it seems to be pretty slower when it runs inside a sub-directory with about four hundred thousand small files (400000), there are others sub-directories inside this sub-directory with others thousands small files.

When tar starts to pack these sub-directories we can see each file reported by tar one by one taking about 1 to 2 seconds among them, its incredibly slow considering small files like a few bytes or dozens KBytes of size.

This filesystem is using jfs2 and it's hosted by an AIX 7.1 system. It's stored in a storage system using some sort of redundancy RAID mode based on SSD devices ("solid state hard disks"). There aren't alerts or any kind of reported issues in this storage system.

A lot of tests were made, sending and not sending the tar packing to either a tape device or a regular file, but the following test is enough to observe that unexpected slowness:

tar -cvf /dev/null .

How jfs2 works to deal with many small files ? Is it something that can be worked around setting up a jfs2 filesystem ? How tar deals with this kind of files structure ?

EDIT: Concurrency info

Tests were also made running it concurrently and not concurrently with others services in this FS or in the whole system. But it doesn't change the slowness, every time the tar command starts to deal with the sub-directories with many files, it behaves slower than before, ever.

Luciano
  • 1,129
  • 4
  • 15
  • 26
  • I really wonder if this is an issue with `tar` or just an issue with _anything_ that tries to iterate over all files in a directory with hundreds of thousands of files? – Kusalananda Aug 30 '17 at 13:35
  • @Kusalananda are you suggesting an *issue* with `opendir()/readir()` api ? We don't know if AIX `tar` implementation uses that, but probably it does. Right ? – Luciano Aug 30 '17 at 13:59
  • 2
    Yes, any directory with a huge amount of files will be slow to work with, on most filesystems. We have a recommendation at work to try to generate no more than a thousand output files from jobs in any one directory. We usually use hashing to distribute files in one or several layers of subdirectories (this may easily be done by taking pieces of, e.g., the MD5 checksum of a filename). – Kusalananda Aug 30 '17 at 14:02

0 Answers0