3

On a Supermicro A2SDi-8C+-HLN4F Atom C3758 system I see that VT-d support is enabled in the BIOS but I can't find any IOMMU groups under Linux (Fedora 31):

$ ls /sys/kernel/iommu_groups
$

The kernel logs seem to indicate that VT-d is detected:

# journalctl -k | grep 'DMAR\|iommu' | sed 's/^.*kernel: //'
ACPI: DMAR 0x000000007E1629C0 000070 (v01 INTEL  BDW      00000001 INTL 00000001)
DMAR: Host address width 39
DMAR: DRHD base: 0x000000fed90000 flags: 0x1
DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
DMAR: RMRR base: 0x0000007f0fa000 end: 0x0000007f10afff
DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
DMAR-IR: HPET id 0 under DRHD base 0xfed90000
DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
DMAR-IR: Enabled IRQ remapping in x2apic mode
iommu: Default domain type: Translated

Do I need to configure something such that I also get IOMMU groups?

Background: I'm looking into this because I can't bind an ethernet device to vfio-pci for dpdk:

# /usr/share/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci 07:00.0
Error: bind failed for 0000:07:00.0 - Cannot bind to driver vfio-pci
# /usr/share/dpdk/usertools/dpdk-devbind.py --status        

Network devices using kernel driver
===================================
0000:05:00.0 'Ethernet Connection X553 1GbE 15e4' if=eno1 drv=ixgbe unused=vfio-pci *Active*
0000:05:00.1 'Ethernet Connection X553 1GbE 15e4' if=eno2 drv=ixgbe unused=vfio-pci 
0000:07:00.1 'Ethernet Connection X553 1GbE 15e4' if=eno4 drv=ixgbe unused=vfio-pci 

Other Network devices
=====================
0000:07:00.0 'Ethernet Connection X553 1GbE 15e4' unused=ixgbe,vfio-pci
# dmesg | tail -n 3
[  +0.169963] ixgbe 0000:07:00.0: complete
[  +0.000561] vfio-pci: probe of 0000:07:00.0 failed with error -22
[  +0.000040] vfio-pci: probe of 0000:07:00.0 failed with error -22
muru
  • 69,900
  • 13
  • 192
  • 292
maxschlepzig
  • 56,316
  • 50
  • 205
  • 279

1 Answers1

2

You need to add intel_iommu=on to kernel load parameters. By default this parameter is disabled (I don't know distro which set this parameter to on by default).

When you've enabled IOMMU in BIOS, then ACPI table translates to kernel DMARs and kernel init this duruing the boot.

When you've set intel_iommu=on, then kernel divide system hardware resources into IOMMU groups. It's quantity and granularity depends on hardware. See VFIO - "Virtual Function I/O" for details.

Yurij Goncharuk
  • 4,177
  • 2
  • 19
  • 36
  • Can confirm, Fedora has the kernel option `CONFIG_INTEL_IOMMU_DEFAULT_ON` disabled, thus IOMMU is disabled by default. Thus, after booting with `intel_iommu=on` the `/sys/kernel/iommu_groups` directory is populated and the `dpdk-devbind.py --bind=vfio-pci ` call succeeds. – maxschlepzig Jun 27 '20 at 15:28