8

Formatting xfs volumes on ubuntu 16.04 is extremely slow in our Virtualbox hypervisor, but not vms running inside Nutanix.

Virtualbox
100 GB => seconds
2TB => seconds

Nutanix (HyperConverged)
100 GB => 4 minutes
2TB => 30+ minutes 

parted -l -s | grep "Error: * unrecognised disk label"
Error: /dev/sdg: unrecognised disk label

parted /dev/sdg mklabel gpt
Information: You may need to update /etc/fstab.

parted -- /dev/sdg mkpart primary xfs 1 -1
Information: You may need to update /etc/fstab.

time mkfs.xfs /dev/sdg1
meta-data=/dev/sdg1              isize=512    agcount=4, agsize=6553472 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0
data     =                       bsize=4096   blocks=26213888, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=12799, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

real     4m7.653s
user    0m0.004s
sys       0m0.028s

Why does it take so long in one hypervisor to format a drive with mkfs, whereas on the other it is nearly instant?

spuder
  • 17,643
  • 36
  • 91
  • 119

4 Answers4

15

This is due to the fact that the hyperconverged hypervisor uses SSD's. The mkfs command formats with NODISCARD (also known as TRIM) by default.

To run mkfs without trim, use the -K option on XFS and -E nodiscard on ext4

XFS

mkfs.xfs -K /dev/sdx

EXT4

mkfs.ext4 -E nodiscard

Warning: Only use -K or -E on new volumes with no existing data.

Using the -K or -E options on drives with existing data, will cause the space to be wasted until the data is overwritten.

spuder
  • 17,643
  • 36
  • 91
  • 119
  • this answer made a big difference creating filesystems on two SSD devices in Raid0 on AWS i3.4xlarge instances - 1hr vs 1sec – jdsumsion Feb 01 '18 at 17:53
  • You do pay a price for this if the underlying storage is a locally attached SSD, because the SSD's internal garbage collection will be slower if it has to deal with a fuller (less-TRIM'ed) disk. This can cause very large latency spikes to happen when the SSD becomes fairly full. – Dan Aug 13 '21 at 22:37
  • @Dan Several modern distributions now run `fstrim` on a weekly schedule (crontab or systemd timer), so the TRIM might still happen, just later. – telcoM Sep 16 '22 at 03:17
  • Yeah, but IO performance will still randomly get slower when that happens. Overall trimming upfront in this scenario is preferable unless you don’t care about IO performance for the VM. – Dan Sep 17 '22 at 14:39
1

From the Red Hat official course (Red Hat System Administration II - RH134), when formating a VDO volume with mkfs.xfs as follow:

# mkfs.xfs -K /dev/mapper/vdo1

It is specified:

"The -K option in the preceding mkfs.xfs command prevents the unused blocks in the file system from being discarded immediately which lets the command return faster."

Pozinux
  • 1,305
  • 5
  • 16
  • 27
0

mkfs -t xfs /dev/md6 this command takes FOREVER on Ubuntu and is real quick on RHEL.

mkfs.xfs -K /dev/sdx seemed to be very quick... but then it looks like it only outputs information.. better to use the time command to see if some work was actually done.

time mkfs.xfs -K /dev/md6 mkfs.xfs: /dev/md6 appears to contain an existing filesystem (xfs). mkfs.xfs: Use the -f option to force overwrite.

real 0m0.005s user 0m0.000s sys 0m0.000s

time mkfs.xfs -K /dev/md6 -f meta-data=/dev/md6 isize=512 agcount=32, agsize=24416912 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0 data = bsize=4096 blocks=781340832, imaxpct=5 = sunit=16 swidth=32 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=381520, version=2 = sectsz=512 sunit=16 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0

real 0m0.572s user 0m0.000s sys 0m0.188s

prabha101
  • 9
  • 1
0

I had the same issue with RHEL 7.9 , if you use external storage SAN with Mpath , for SAN if you use BOND or NIC teaming mkfs.xfx extremely slow I have reinstalled the OS twice with a different version RHEL 7.5 same, I have multiple RHEL servers in production with the same setup but without NIC teaming (round-robin) I have removed in the new (problem) server I have tried it works perfectly. I presume When I execute the mkfs command it discard blocks for LUN the execution happens round-robin the connection was keep changing to the external storage so it extremely slow. I have spent week and i googled I checked in RHEL support portal but no luck, finally I have removed nic teaming and its works.

PRAKASH
  • 1
  • 1