11

enter image description here

I've read that the color red indicates "kernel processes." Does that mean little daemons that are regulating which task gets to use the CPU? And by extension, transaction costs in an oversubscribed system?

I'm running some large-scale geoprocessing jobs, and I've got two scripts running in parallel at the same time.

The first script does the actual processing, on all 96 cores. It is responsible for almost all of the memory use.

The second script uses curl to download the data to feed the first process, and it does so in parallel. I wrote it to download only until there are n_cores * 3 files downloaded. If that constraint isn't met, it waits for a minute or so and then check again. So, most of the time it isn't running -- or rather it is executing the Sys.sleep() command in R.

I've experimented with using fewer cores for the downloading process. When I do so, it can't keep up with the processing script (I'm DLing from S3).

TL;DR: Would my processes run faster if I could make htop less red? And are they red because there are more processes than cores?

generic_user
  • 597
  • 2
  • 5
  • 10
  • What kind of processing does your script do? The "red" could indicate I/O time, if I'm not mistaken, so it would be interesting to know whether you consider it normal for your script to do lots of it. Either way, others may recommend better tools than htop to find out. (perf comes to mind.) – dhag Feb 20 '18 at 16:40
  • Does reading from netcdfs count as I/O? If so then yes. Will look into perf. – generic_user Feb 20 '18 at 17:40
  • 3
    I found this post by looking up "htop" in Google Images and... can I just say, holy cow... I want your machine ._. – Jack Bauer Apr 27 '19 at 22:05
  • @Jack Bauer Amazon rents them – generic_user Apr 28 '19 at 00:29
  • @generic_user I assumed it was some such scenario. Still makes me jealous :p – Jack Bauer Apr 29 '19 at 05:04

1 Answers1

16

Red represents the time spent in the kernel, typically processing system calls on behalf of processes. This includes time spent on I/O. There’s no point in trying to reduce it just for the sake of reducing it, because it’s not time that’s wasted — it’s time that’s spent by the kernel doing useful stuff (as long as you’re not thrashing, so look at the number of context switches etc.).

I've experimented with using fewer cores for the downloading process. When I do so, it can't keep up with the processing script (I'm DLing from S3).

suggests that your current setup is evenly balanced between the I/O needed to feed the processing, and the processing itself, which is a rather nice result. If you suspect that you’ve got too many processes running, and that that’s causing waste (by thrashing), then you could try reducing the number of geoprocessing jobs, to see if your overall throughput increases. The usual benchmarking tips apply: identify what you’re going to tweak, determine what resulting variations could occur and what they mean, only tweak one thing at a time, and measure everything.

Stephen Kitt
  • 411,918
  • 54
  • 1,065
  • 1,164
  • Thanks a lot! Indeed, my R script makes lots of calls to `system("some bash command")`. Until you answered my question, I assumed that those processes would come out green, instead of red, on htop. I bet that R's netcdf libraries also come out red. You also introduced me to the term "thrashing", which I hadn't come across in this context before. – generic_user Feb 20 '18 at 18:12
  • 1
    @generic_user FYI, system calls in this context are functions exposed by the kernel to userspace. These are the fundamental operations that any program can perform, such as `sys_read` or `sys_open`. R's `system()` function doesn't make a system call, rather, executes another userspace process – Outurnate Feb 20 '18 at 21:53