A follow-up from this question.
My further reading on Docker storage drivers revealed that the overlay driver merges all the image layers into lower layers using a hard link implementation which cause excessive inode utilization. Can someone explain this? As far as I know, creating hard links does not create a new inode.
OverlayFS is a union filesystem, and there are two storage drivers at the Docker level that make use of it: the original/older version named overlay and the newer version named overlay2. In OverlayFS, there is a lower-level directory which is exposed as read-only. On top of this directory is the upper-level directory, which allows read-write access. Each of these directories is called a layer. The combined view of both the lower-level and upper-level directories is presented as a single unit, called the 'merged' directory.
The newer overlay2 storage driver natively supports up to 128 such layers. The older overlay driver is only able to work with two layers at a time. Since most Docker images are built using multiple layers, this limitation is fairly significant. To work around this limitation, each layer is implemented as a separate directory that simulates a complete image.
To examine the differences on my test system, I pulled the 'ubuntu' image from Docker Hub and examined the differences in directory structure between the overlay2 and overlay drivers:
[root@testvm1 overlay2]$ ls */diff
4864f14e58c1d6d5e7904449882b9369c0c0d5e1347b8d6faa7f40dafcc9d231/diff:
run
4abcfa714b4de6a7f1dd092070b1e109e8650a7a9f9900b6d4c3a7ca441b8780/diff:
var
a58c4e78232ff36b2903ecaab2ec288a092e6fc55a694e5e2d7822bf98d2c214/diff:
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
c3f1a237c46ed330a2fd05ab2a0b6dcc17ad08686bd8dc49ecfada8d85b93a00/diff:
etc sbin usr var
[root@testvm1 overlay]# ls */root/
001311c618ad7b94d4dc9586f26e421906e7ebf5c28996463a355abcdcd501bf/root/:
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
048f81f400f7d74f969c4fdaff6553c782d12c04890ad869d75313505c868fbc/root/:
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
8060f0c647f24050e1a4bff71096ffdf9665bff26e6187add87ecb8a18532af9/root/:
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
fbdef944657234468ee55b12c7910aa495d13936417f9eb905cdc39a40fb5361/root/:
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
In the overlay representation, each layer simulates a complete image, while the overlay2 layers only contain the exact differences between layers. In the overlay driver's approach, hard links are used as a way to save space between the different layers. However, this method is still not perfect, and new inodes are required when the image data contains special files such as symbolic links and character devices. This can quickly add up to large number of inodes.
The inode distribution betwen the overlay2 and overlay drivers on my test system are as shown below.
[root@testvm1 overlay2]$ du --inodes -s *
8 4864f14e58c1d6d5e7904449882b9369c0c0d5e1347b8d6faa7f40dafcc9d231
27 4abcfa714b4de6a7f1dd092070b1e109e8650a7a9f9900b6d4c3a7ca441b8780
3311 a58c4e78232ff36b2903ecaab2ec288a092e6fc55a694e5e2d7822bf98d2c214
1 backingFsBlockDev
25 c3f1a237c46ed330a2fd05ab2a0b6dcc17ad08686bd8dc49ecfada8d85b93a00
5 l
[root@testvm1 overlay]# du --inodes -s *
3298 001311c618ad7b94d4dc9586f26e421906e7ebf5c28996463a355abcdcd501bf
783 048f81f400f7d74f969c4fdaff6553c782d12c04890ad869d75313505c868fbc
768 8060f0c647f24050e1a4bff71096ffdf9665bff26e6187add87ecb8a18532af9
765 fbdef944657234468ee55b12c7910aa495d13936417f9eb905cdc39a40fb5361
The total count of inodes on overlay2 comes to 3378 on my system. Using overlay, this count goes up to 5615. This value is considering a single image and with no containers running, so a large system with a number of docker containers and images could quickly hit the inode limit imposed by the backing filesystem (XFS or EXT4, where the /var/lib/docker/overlay directory is located).
Due to this reason, the newer overlay2 storage driver is currently the recommended option for most new installations. The overlay driver is deprecated as of Docker v18.09 and is expected to be removed in a future release.