Q3: What do the results of grep -i nmi /proc/interrupts mean?
It becomes clearer if you see the headings of the columns:
$ cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
NMI: 24 18 21 18 Non-maskable interrupts
This command displays statistics about the interrupts per CPU.
Q2: What's the advantage of disabling nmi_watchdog?
The nmi_watchdog can, under some circumstances, generate a high number of NMI interrupts (i.e. when using local APIC and you have high system load). A high number of interrupts may slow down your system.
Q1: What could happen if I disable nmi_watchdog permanently?
Imagine your system locks up. There's 2 possibilities now:
1) You have a hardware NMI button (some servers do). Push it, then the kernel (if properly configured) dumps a trace to console and reboots.
2) Your kernel reaches a halting state that can't be interrupted by any other method. In this case, the watchdog can reboot the machine.