5

I't trying to move around 4.5 million files (size ranges from 100 - 1000 bytes) from one partition to other. The total size of the folder is ~2.4 GB

First I tried to zip it and move the zipped file to the new location. It is able to paste only ~800k files and shows "out of space" error.

Next I tried the mv command and it also resulted in same condition.

Using rsync also resulted in the same error with only ~800k files being moved.

I checked the disk free status and it is way under the limit. ( The new partition has ~700 GB free space and the required space is ~2.4 GB).

I checked the free inode for that partition it is the same. It is using only ~800k out of the maximum possible 191 M inodes. ( I had actually formatted the partition with 'mkfs.ext4 -T small /dev/sdb3' )

I have no idea of what is going wrong here. Everytime it is only able to copy or move ~800k files only.

deathholes
  • 807
  • 1
  • 6
  • 11
  • 1
    Are all the files in the same directory? You might have to split them up. What are the filesystems involved? – DopeGhoti Jan 10 '18 at 17:07
  • Yes, all of them are in the same directory. The filesystem is ext4 in both the source and destination partition. Why do i need to split them? What difference does it make to split them in directories? – deathholes Jan 10 '18 at 17:59
  • I think DopeGhoti means it can be a max number of separate files on each directory restrictions, therefore I think this limit is higher than that. – francois P Jan 10 '18 at 18:27
  • @deathholes, can you check to see what `tune2fs -l /dev/$device` shows on the devices in question? If there are any differences between the two partitions? – ilkkachu Jan 10 '18 at 18:45
  • 2
    I found the error was due to hash collisions in the ext4 filesystem. Did a 'tune2fs -O "^dir_index" ' on the partition and everything is good to go albeit a little slow. – deathholes Jan 12 '18 at 16:48
  • 1
    @deathholes Note that If you solved the problem yourself you can still answer your own question. – sakisk Jan 12 '18 at 17:42
  • This is a good discovery. Please do write it up for future reference. – roaima Jan 15 '18 at 00:09

1 Answers1

3

I have found the reason for the error (found it on a different forum). The error was due to the hashing algorithm used by ext4 which is enabled by "dir_index" parameter. There were too many hashing collisions for me so I disabled it by the following command:

tune2fs -O "^dir_index" /dev/sdb3

The downside is that my partition is slower than before due to no indexing.

For more information on the problem : ext4: Mysterious “No space left on device”-errors

deathholes
  • 807
  • 1
  • 6
  • 11