11

I pulled up the console on a virtual machine I manage today and was greeted with some kernel messages:

[5912557.130943] Uhhuh. NMI received for unknown reason 20 on CPU 0.
[5912557.131115] Do you have a strange power saving mode enabled?
[5912557.131287] Dazed and confused, but trying to continue
[6064281.393568] Uhhuh. NMI received for unknown reason 30 on CPU 1.
[6064281.393888] Do you have a strange power saving mode enabled?
[6064281.394235] Dazed and confused, but trying to continue

That's just a few of them, both 20 and 30 occur on CPU 0 and 1.

  • VM is Debian Jessie, BIOS boot ("QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014"; kernel 3.16.0-4-amd64)
  • Hypervisor is libvirt/KVM running on Debian testing (currently Debian's 4.7.0-1-amd64; qemu 1:2.7+dfsg-3).
  • Hardware is an Opteron 6344 on a Supermicro H8SGL-F with ECC RAM with scrub enabled.

I don't see any NMI or EDAC error/warning messages on the host.

Any idea what is causing these NMI messages on the guest? Are they anything to worry about?

(May be related to NMI received for unknown reason 20 — Do you have a strange power saving mode enabled? but that appears to be bare metal).

derobert
  • 107,579
  • 20
  • 231
  • 279
  • I wonder wether it would help to pass to the kernel of the VMs `noapic apci=off` – Rui F Ribeiro Dec 01 '16 at 03:40
  • @RuiFRibeiro Well, currently the VM is working without any (apparent) problems. It's in production, so I'd rather not go around rebooting to try random kernel options just to see. Would be a different story if it were to help a kernel dev to debug the problem, etc. (Plus, it's not like they're frequent—it'd take a while to be sure.) – derobert Dec 01 '16 at 17:42
  • I've been trying to track down the same issue for some time. Some data points that may be helpful are: host kernel version, qemu version, whether the VM uses BIOS or UEFI boot, whether the VM uses i440fx or q35. – Michael Hampton Dec 26 '16 at 07:39
  • @MichaelHampton requested details added to the question. – derobert Dec 26 '16 at 16:48
  • I've got the same issue, here are the details (very similar actually): VM is Debian jessie (3.16.0-4-amd64) with BIOS 1.7.5-20140531_083030-gandalf (04/01/2014). Hypervisor is libvirt/KVM on Debian jessie, but with backported kernel (4.7.0-0.bpo.1-amd64). Hypervisor hardware is two Opteron 6272s, with ECC RAM (motherboard currently unknown, but likely Supermicro of some kind). Given these details are remarkably similar to derobert's, I'm not too surprised that I encounter this issue too, but hopefully they help. – jvperrin Dec 31 '16 at 07:36
  • @MichaelHampton another person (jvperrin) has a similar setup and has posted details in the comment above this one. Pinging you because otherwise I doubt you'll be notified. – derobert Dec 31 '16 at 07:42
  • Well, this doesn't appear to be distro-specific, and [it's apparently almost three years old](https://lists.nongnu.org/archive/html/qemu-devel/2014-04/msg00395.html) with still no solution. – Michael Hampton Dec 31 '16 at 07:49
  • @MichaelHampton that is a bit different as no migration is involved here. Nor does it die after the error. – derobert Dec 31 '16 at 08:07
  • There doesn't have to be a migration for the problem to appear, as some of the list messages mentioned. And I wouldn't have brought it up if it were irrelevant. – Michael Hampton Dec 31 '16 at 08:10
  • Did you install the `qemu-guest-utils` package on the VM? It might be that the NMI (Non-Maskable Interrupt) is something qemu is trying to tell the guest OS. If that is the case, then the guest-utils package *should* take care of it. – Wouter Verhelst Jan 02 '17 at 13:08

5 Answers5

3

I had the same issue using a similar setup:

  1. AMD CPU (although I've seen reports of the same issue with Intel CPUs, but none of my hypervisors running on Intel CPUs have this issue, even with CPU passthrough enabled).
  2. Debian, kernel 4.x on the hypervisor and guest (4.9.0-4-amd64 in my case on both).

My solution was to switch my guest VM to use a QEMU emulated CPU rather than CPU passthrough. This entailed removing the <cpu mode='host-passthrough'/> line from the guest definition file.

Update 1: I did further investigation and the troublesome elements were under the clock element:

<clock offset='utc'>
  <timer name='rtc' tickpolicy='catchup'/>
  <timer name='pit' tickpolicy='delay'/>
  <timer name='hpet' present='no'/>
</clock>

The real solution was to remove the three <timer> elements, after which <cpu mode='host-passthrough'/> could be enabled again.

For completeness I added a similar answer to the linked question.

Update 2 As Simon Richter mentioned, this fix may not work for everyone as the settings I've removed are the defaults. If so, his solution should be followed.

mjturner
  • 7,082
  • 1
  • 26
  • 32
2

The problem seems to be that the End of Interrupt isn't communicated properly.

For libvirt, make sure eoi is enabled:

<domain>
  …
  <features>
    <apic eoi='on'/>
    …

On the command line for KVM that translates to

-cpu …,+kvm_pv_eoi

This seems to work for us with -M q35, host cpu passthrough and default config otherwise (RTC interrupts queued, PIT interrupts dropped, HPET unavailable).

Simon Richter
  • 4,409
  • 18
  • 20
0

<acpi eoi="on"/> does not work for me, after some time the problem re-occurs and Uhhuh. NMI received for unknown reason 20 on CPU 0 are printed again by the Linux kernel.

During my further research I found two commits fixing bugs in the iTCO_wdt (Intel Total Cost of Ownership Watch-Dog Timer) Qemu emulation:

Debian-9-Stretch only had qemu-2.8, which does not contain those patches. Debian-10-Buster has qemu-3.1, where I did not observe the problem.

The driver can be added to the deny-list to prevent it from getting loaded:

echo blacklist iTCO_wdt >/etc/modprobe.d/itco-wdt.conf

After that make sure to shutdown and re-start the VM to kill the old Qemu process and get a new instance. A reboot re-uses the old Qemu process, which would preserves the broken state inside Qemu. But a reboot is required to trigger the bug as the bug never happend for me on 1st boot, but only after a reboot or a suspend-to-disk and resume cycle.

pmhahn
  • 126
  • 1
  • 4
-1

I have had the same problem on Debian 9 and Qemu 2.8.1(Debian 1:2.8+dfsg-6+deb9u5).
I resolved it by replace videocard model from virtio to cirrus (or you can try use another model from qemu man page).

Oleg Golovanov
  • 1
-1

I had the same problem on Debian 11 hypervisor after cloning Debian 10, 11 VMs from another hypervisor. It helped me to restart VMs and set CPU Model value Clear CPU Configuration.

Oleg Kondaurov
  • 1