5

How do I transition an arbitrary PCIe device from the D0 state (powered on) to the D3cold state (no current to the device) using common utilities on a Linux system? Presumably this involves writing to a sysfs file, but other methods that may be more involved are welcome too, as long as they can be done in userspace. This question is based on another one regarding power cycling.

I am specifically looking for a hardware-agnostic (x86 at least) way to put a device into D3cold.

Assume that /sys/bus/pci/devices/*/d3cold_allowed is 1.

Note: I'm aware that a PCI function can't be put into D3cold on its own, so I suppose a better way to phrase this would be "How can I put a specific PCIe device's bus into the L3 state?".

forest
  • 2,585
  • 15
  • 27
  • The PCIe spec does not speak to this as D3cold is simply turning off power to the card. Putting a single card in D3cold will be motherboard dependent and may not be possible in general - the motherboard would have to provide a way to remove power from individual PCIe slots, which I do not believe is a standard feature. – alex.forencich Apr 10 '19 at 05:56
  • Well, I guess you can send all of the PCI cards into D3cold by simply turning the machine off. But presumably this is not what you want to do. – alex.forencich Apr 10 '19 at 05:58
  • Related, but I don't believe the answers are correct: https://stackoverflow.com/q/28078711/9782169 – forest Apr 02 '22 at 22:40
  • I think d3cold requires support from the motherboard in terms of actually turning off power to the card. So if the motherboard doesn't support it, then you can't do it, and hence there may be no way to do it in a device-independent way. – alex.forencich Apr 02 '22 at 22:58
  • @alex.forencich How would it be done if the motherboard does support it? Some posts claim that it can be done by writing 0 to `/sys/bus/pci/slots//power`. Is that not the case? – forest Apr 02 '22 at 23:00
  • I don't know, I have never seen those sysfs entries on any of the machines I have checked. It's possible that it's only supported for things like laptop GPUs. – alex.forencich Apr 02 '22 at 23:13
  • Okay, so one of the machines I checked has a "power" entry on one of the slots. The other two machines I checked had none. However, I did find https://docs.kernel.org/power/pci.html and https://github.com/Bumblebee-Project/bbswitch/blob/master/bbswitch.c . So it seems like from the kernel, you can call pci_set_power_state(dis_dev, PCI_D3cold) which will then use pci_platform_power_transition(dev, state) to turn off power via ACPI, falling back to D3hot if this isn't supported. Not sure how to do the same from userspace. – alex.forencich Apr 03 '22 at 07:33
  • @alex.forencich Yeah I saw `pci_set_power_state()` in the code as well but couldn't find out how to trigger it from userspace either. Did the `power` entry trigger that on your machine? – forest Apr 03 '22 at 20:33
  • Didn't try it. From my reading of the docs, it seems like the driver is supposed to be responsible for the power state. It would probably cause trouble to change the power state out from under the driver, so I suspect there may not be a way to do this from userspace. Looking at bbswitch, it makes sure the driver isn't loaded, and then it has to jump through a few hoops with respect to suspend and such. And it also appears that it will fall back on D3hot even if D3cold is requested, if the firmware does not support powering off the slot. – alex.forencich Apr 03 '22 at 21:27
  • 1
    Check this forum thread, it solved my problem: https://forums.gentoo.org/viewtopic-t-1089466-start-0.html – Anthony Apr 21 '22 at 12:18
  • @tosh That put it into D3hot, at least. Thanks! – forest Apr 21 '22 at 20:17
  • You're welcome ^_^ – Anthony Apr 21 '22 at 20:29
  • Seems like vfio-pci only supports D3hot by now: https://lore.kernel.org/lkml/[email protected]/T/ – Anthony Apr 22 '22 at 13:43

0 Answers0