You can’t, for a number of reasons.
The first is that a directory’s size grows, but it doesn’t shrink (on most file systems anyway). Try this:
mkdir testdir && cd testdir
touch {1..100000}
rm {1..100000}
ls -ld ../testdir
This will produce a fairly large directory (nothing like yours admittedly, but that’s irrelevant here) containing no files...
The second is that in most cases, file records inside a directory entry are variable in length, depending on the file’s name. See for example the ext4 disk layout.
The third is that the directory might not even be linear, which complicates matters further.
The fourth is that a directory’s size is a multiple of the block size, so a directory with one file and a directory with twenty will usually have the same size.