9

I'm running a test program which generates a large number of threads and asynchronous I/O. I'm seeing very high counts of these interrupts in /proc/interrupts, the program cannot scale beyond a certain point because CPUs are 100% saturated with softirq processing.

According to: http://www.kernel.org/doc/man-pages/online/pages/man5/proc.5.html CAL stands for 'remote function call interrupt' but that's all the info I can find on google. So...what does that mean? I have smp_affinity assigned for my I/O adapter cards and those are not the CPUs taking a huge number of CAL interrupts. Is it a sign my program is running with incorrect SMP affinity?

twblamer
  • 919
  • 2
  • 10
  • 19
  • 1
    Without seeing the source code of the program, or at least a simplified pseudocode example, it is difficult to say what is wrong. Also what do you mean by "asynchronous IO"? If you mean posix aio, then the whole point is to be able to queue a bunch of background IO without having to create a bunch of threads. The more threads you create, the more overhead there will be, so how many threads are we talking about here? – psusi Apr 30 '12 at 23:37
  • I'm using a recent version of fio [http://freecode.com/projects/fio], configured to run ~20 "jobs" at the same time, using libaio to keep multiple random I/Os queued up to each drive. To be honest I'm not sure how many threads it's spawning. I did notice that when I disabled my second CPU socket (leaving only socket 1 enabled), I got similar I/O performance from fio but the number of CAL interrupts dropped dramatically. So maybe it's due to fio threads running on socket 2, invoking interrupt handlers that are affinitized to socket 1? I'm wondering if I can do some kind of trace or profiling. – twblamer May 01 '12 at 20:14

1 Answers1

2

Remote function call interrupts belong to the family of Inter-Processor Interrupts, that is interrupt signals sent by one processor to any other processor in the system and delivered not through an IRQ line, but directly as a message on the bus that connects the local APIC of all CPUs.

Remote function call interrupts are generally sent to all processors but the sender, forcing those to run a function passed by the sender. Under a linux x86 kernel :

The corresponding interrupt handler is named call_function_interrupt( ).
The interrupt is usually sent by means of the smp_call_function( ) facility function.

Typical use cases include stopping all cpus when shutting down, calibrating clocks, flushing TLBs… and more.

Of course different smp-affinity settings can indirectly impact on the number of TLBs flushes but because this impact is indirect and because TLB's flushes are not the only reason for function call interrupts, their rate should not be used at first and alone to judge the efficiency or the affinity settings.

First judge of pertinence for smp-affinity settings is the number of context switches. The lower, the better.

MC68020
  • 6,281
  • 2
  • 13
  • 44