3

I migrated some special directories (not supported by common migration tools) from one Synology NAS volume (ext4) to another (btrfs). After checking if the synchronization was successful I found that sizes differ a lot.

I understand that there are block size differences but du gives me wrong data size within the same volume.

The total size of the whole directory is more than 1 TB so I narrowed the commands below to a smaller sub-directory.

Different sizes:

sudo du -sm /volume[12]/@synologydrive/@sync/repo/1/2
11418   /volume1/@synologydrive/@sync/repo/1/2
11122   /volume2/@synologydrive/@sync/repo/1/2

But for each sub-directory I get correct sizes (more or less when bytes are used) - sub-directory n is selected:

sudo du -sm /volume[12]/@synologydrive/@sync/repo/1/2/n
295     /volume1/@synologydrive/@sync/repo/1/2/n
295     /volume2/@synologydrive/@sync/repo/1/2/n

sudo du -sb /volume[12]/@synologydrive/@sync/repo/1/2/n
308387853       /volume1/@synologydrive/@sync/repo/1/2/n
308391693       /volume2/@synologydrive/@sync/repo/1/2/n

But, I'm getting completely different sizes when * is used (i.e. even on the same volume compare command above and this one on volume2):

sudo du -sm /volume[12]/@synologydrive/@sync/repo/1/2/* | grep n$
295     /volume1/@synologydrive/@sync/repo/1/2/n
200     /volume2/@synologydrive/@sync/repo/1/2/n

sudo du -sb /volume[12]/@synologydrive/@sync/repo/1/2/* | grep n$
308387853       /volume1/@synologydrive/@sync/repo/1/2/n
209533219       /volume2/@synologydrive/@sync/repo/1/2/n

I also counted sizes of all files under the n directory and I'm getting the same sizes:

ls -lA /volume1/@synologydrive/@sync/repo/1/2/n | tr -s ' ' | cut -f5 -d" " | awk '{s+=$1} END {print s}'
308387597
ls -lA /volume2/@synologydrive/@sync/repo/1/2/n | tr -s ' ' | cut -f5 -d" " | awk '{s+=$1} END {print s}'
308387597

So, it seems that the directories are "rsynced" correctly (same number of files, same sizes) and tried to exclude differences between two different file systems. But du gives me significantly bigger size on new volume1 (or more exactly - significantly smaller on old volume2).

Do you have any explanation for it?

Notes:

  • volume1 is new btrfs target volume (I copied data to)
  • volume2 is old ext4 source volume (I copied data from)
  • Data were copied using sudo rsync -a --progress --delete /volume2/@synologydrive /volume1
CraZ
  • 143
  • 8
  • 2
    Do you have sparse files and/or hard-linked files? Ho did you run the `rsync` command? Did you include `--sparse` and `--hard-links`? It also matters how you count the sizes that you compare. For sparse files, you either count the apparent sizes (shown by `ls` by default), or you count the actual size (number of bytes on disk). For hard-linked files `du` would only count a file once. – Kusalananda Dec 02 '21 at 11:36
  • @they, `rsync` command is mentioned above in Notes. There are no hard-links (AFAIK). I also tried `rsync` with `--sparse` switch but no change. However, the core of the question is probably not in rsync itself as the sizes differ on the source volume - there are differences between `du ... /1/2/n` and `du ... /1/2/n/*`. – CraZ Dec 02 '21 at 11:48
  • @they, well, your point with hard-links was really good indeed! I've checked and there are hard-links in in the source directory (thousands of hard links!). And if I used `du -l` switch I'm getting same folder sizes. Thanks, I've learned some new today ;-) – CraZ Dec 02 '21 at 12:07

1 Answers1

1

Thanks to @they in the comments under OP, I've found the root cause. Original source volume contained lots of hard-linked directories/files and they all were copied to new volume as new standalone (fixed) directories/files.

There were actually two things that helped me to understand the problem:

  • rsync doesn't respect hard links by default. Lessons learned: run rsync with --hard-links switch, potentially also with --sparse switch to handle smaller files efficiently.

  • du doesn't include size of hard-linked files by default. Lessons learned: run du -l switch to count size of all files, incl. those hard-linked. Similarly to sizes what ls displays.

Without knowing these two aspects and I wouldn't know if my files are synced correctly.

CraZ
  • 143
  • 8