5

kworker process consumes 75% of one CPU. The offending kworker thread is related with ACPI stuff:

sudo cat /proc/THE_PID_OF_KWORKER_PROCESS/stack
[<ffffffff85c0c705>] acpi_ns_evaluate+0x1bc/0x23a
[<ffffffff85bffe09>] acpi_ev_asynch_execute_gpe_method+0x98/0xff
[<ffffffff85be4e30>] acpi_os_execute_deferred+0x10/0x20
[<ffffffff8588dc21>] process_one_work+0x181/0x370
[<ffffffff8588de5d>] worker_thread+0x4d/0x3a0
[<ffffffff85893f1c>] kthread+0xfc/0x130
[<ffffffff8588de10>] process_one_work+0x370/0x370
[<ffffffff85893e20>] kthread_create_on_node+0x70/0x70
[<ffffffff858791ba>] do_group_exit+0x3a/0xa0
[<ffffffff85e6a2b5>] ret_from_fork+0x25/0x30
[<ffffffffffffffff>] 0xffffffffffffffff

so I started debugging by rebooting with some acpi related kernel parameters, such as: 

acpi=off : Completely solves the high cpu usage, but computer no longer suspends. 
acpi=ht : no effect, still high cpu usage
pci=noacpi : not booting at all
pnpacpi=off : no effect, still high cpu usage
noapic : worse, 100% cpu usage 
nolapic : worse, 100% cpu usage

uname -a: Linux 4.13.0-1-amd64 #1 SMP Debian 4.13.4-1 (2017-10-01) x86_64 GNU/Linux

My root folder disk layout is: BTRFS over LVM over LUKS.

How can I find the root of the problem?

Update

I wasn't using my external hard drive which uses a DVD enclosure to be attached to the laptop.

enter image description here Today I reconnected the drive and kworker consumed that excessive amount of CPU again. Note that I didn't mount partition from the external drive, only attaching caused that cpu usage.

ceremcem
  • 2,231
  • 1
  • 23
  • 53

4 Answers4

12

When I checked ACPI interrupts, I noticed that gpe6F had a very high trigger count:

root@HOST:~# grep . -r /sys/firmware/acpi/interrupts/

/sys/firmware/acpi/interrupts/ff_gbl_lock: 0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe15:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe43:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe71:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe05:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe33:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6D:       0         disabled     unmasked
/sys/firmware/acpi/interrupts/gpe61:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe2F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe23:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe5D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe51:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe13:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe41:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe03:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe31:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe21:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe78:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe11:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/ff_pwr_btn:  0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/ff_slp_btn:  0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe68:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe01:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/ff_pmtimer:  0     STS invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe58:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe48:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe76:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe38:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe66:       4  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe28:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe56:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe18:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe46:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe74:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe08:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/sci:  819678
/sys/firmware/acpi/interrupts/gpe36:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe64:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe26:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/error:       0
/sys/firmware/acpi/interrupts/gpe54:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe16:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe44:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe72:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe06:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/ff_rt_clk:             disabled     unmasked
/sys/firmware/acpi/interrupts/gpe34:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6E:  262969  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe62:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe24:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe5E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe52:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe14:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe42:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe70:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe04:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe32:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe60:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe22:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe50:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe79:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe12:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe40:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe69:       0         disabled     unmasked
/sys/firmware/acpi/interrupts/gpe02:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe30:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe59:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe20:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe49:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe77:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe10:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe39:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe67:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe00:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe29:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe57:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe19:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe47:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe75:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe09:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe37:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe65:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe27:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe55:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe17:       0     STS invalid      unmasked
/sys/firmware/acpi/interrupts/gpe45:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/sci_not:     101
/sys/firmware/acpi/interrupts/gpe73:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe07:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe35:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6F:  560719     STS enabled      unmasked
/sys/firmware/acpi/interrupts/gpe63:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe25:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe_all:  823692
/sys/firmware/acpi/interrupts/gpe53:       0         invalid      unmasked

I disabled it:

root@HOST:~# echo "disable" > /sys/firmware/acpi/interrupts/gpe6F

and everything went back to normal:

Linux 4.9.0-6-amd64 (HOST)  05/01/2018  _x86_64_    (4 CPU)

12:30:27 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:30:30 PM     all      6.88      0.00      1.26      0.17      0.00     91.69
12:30:33 PM     all      6.45      0.00      1.17      0.17      0.00     92.20
12:30:36 PM     all      7.15      0.00      1.01      0.34      0.00     91.51
Average:        all      6.83      0.00      1.15      0.22      0.00     91.80
Michael Mrozek
  • 91,316
  • 38
  • 238
  • 232
DEBFIX
  • 121
  • 1
  • 3
  • 4
    Could you please explain what you are doing here and why? Also it is difficult to see what of the information you have posted is important. – Raphael Ahrens May 01 '18 at 12:22
  • 1
    For me it was `gpe61`. I wish I knew what these were, but my idle CPU usage went down from 11-15% to around 1%. – TheWanderer Jun 16 '18 at 04:40
  • Thanks! This solved the issue for me. Should be accepted as best answer – Kyrol Jun 10 '19 at 09:33
  • This solved it for me as well. I had these 2.5' sata HDDs connected via USB C to SATA adapter bought from Amazon. Whenever it's connected one CPU core went to 100% utilised by kworker. Thanks! – retromuz Sep 04 '19 at 09:23
  • Same interrupt here: `gpe6F` (Manjaro, 3 different drives). It would be nice if some kernel hacker could tell us what this int is for, or how could we possibly dig deeper (is it from a specific module / device?). For now I'm enyoing the extra CPU core I got back. – Tomasz Gandor May 21 '20 at 23:07
2

kworker is used by encryption (LUKS) so you will see it consume CPU on disk IO (which can be background FS maintenance). You should run :

cryptsetup benchmark

to see if you are missing hardware acceleration.

user1133275
  • 5,488
  • 1
  • 19
  • 37
  • How can I interpret [the results](https://gist.github.com/ceremcem/3736cb684291df9c333539ce164ec1fd)? – ceremcem Oct 17 '17 at 22:12
  • @ceremcem a small aes-xts 512b comparison : https://forum.odroid.com/viewtopic.php?f=55&t=18002&start=200#p193600 if your numbers for that sub test are in the low end you can expect slowness/high cpu use. – user1133275 Oct 18 '17 at 06:22
  • The greatest score in the list is `1393 MB/s` and [mine is](https://gist.github.com/ceremcem/3736cb684291df9c333539ce164ec1fd#file-cryptsetup-benchmark-txt-L18) `1494 MB/s`. So this is not the encryption case I suppose. – ceremcem Oct 18 '17 at 07:44
  • @ceremcem it still might be depending on IO, you could unmount all but a small partition, stop all services, to see if the issue persists.... or start checking logs maybe 'iotop -o -b -n 1' – user1133275 Oct 18 '17 at 07:55
  • Coincidentally I needed to remove my expansion hard disk from the laptop which was recently installed in place of DVD drive and installed the DVD drive temporarily. Now there is no `kworker` consuming that CPU resource. I'll try somethings more and let you know the results. – ceremcem Nov 01 '17 at 10:43
  • I'm having a similar issue in AWS EC2 but the installed version of `cryptsetup` doesn't support a "benchmark" option. I'm running v1.5.1 on this instance. Is this option only support by newer versions? – SaxDaddy Jan 09 '18 at 01:24
2

I had a similar problem running Lubuntu 20.10, 5.8 kernel, in an old Macbook 2006. In my case the solution was to mask gpe17 using boot parameter

I added boot parameter acpi_mask_gpe=0x17 to GRUB and rebooted.

Add acpi_mask_gpe=0x17 to /etc/default/grub line GRUB_CMDLINE_LINUX_DEFAULT, then run update-grub and reboot.

That solved the high CPU utilization.

I found the problem with this command: grep . -r /sys/firmware/acpi/interrupts/

The issue: /sys/firmware/acpi/interrupts/gpe17: 3491042 EN enabled unmasked

The solution was provided by Peter Nowee in his answer: Disable GPE ACPI interrupts on boot?

ofbarea
  • 61
  • 2
0

Apparently it was a hardware related problem. The kworker process was related with the external hdd enclosure where the enclosure had a switch to use for different brands:

enter image description here

Toggling the switch solved the issue.

ceremcem
  • 2,231
  • 1
  • 23
  • 53