2

I have monit configured to terminate a pair of processes when they consume too much CPU, but once either is terminated, monit summary reports "Not monitored" for the process, even after the OS relaunches it.

How do I configure to 'monit' monitor a process after it restarts?

check process soagent
    matching "soagent"
    stop program = "/usr/bin/pkill -9 soagent"
    if cpu > 20% for 1 cycles then stop
    if cpu > 20% for 1 cycles then alert

check process callservicesd
    matching "callservicesd"
    stop program = "/usr/bin/pkill -9 callservicesd"
    if cpu > 20% for 1 cycles then stop
    if cpu > 20% for 1 cycles then alert

macOS 10.13.4, monit 5.25.2

orome
  • 416
  • 1
  • 5
  • 15

1 Answers1

2

The issue is that you're only telling it to stop the process, which stops monitoring it too.

In this case, the easiest solution is to write a script that will tell launchd to restart the process (I unfortunately can't help with that specifically, I don't know much about macOS), and then use a single conditional in the monit config for each service to run the script like this:

if cpu > 20% for 1 cycles then exec "/path/to/script"

The exec clause will still send an email alert when it triggers, the script will get run to restart the process/service, and then monit will keep watching it since it's not been told to stop watching it.

Assuming you take this approach, you can also remove the stop program line from the definitions too, as that won't be used unless you run monit stop on the services.

Edit to clarify things further:

Monit expects that the OS isn't handling process supervision. In other words, it assumes it's the only thing responsible for starting or stopping processes, and therefore when you tell it to stop something, it will both stop that process, and stop monitoring it, because it assumes nothing will try to start it again (and thus it's a bit of a pain to use at times with macOS's launchd or systemd on Linux).

In your case, what you want is to restart the process if it uses too much processing power and let you know when this happens. There are two ways you can do this with monit:

  1. Define a start program and stop program for it, which if using a separate process supervisor as your init system (which macOS does) should be commands that tell that to start or stop the program, and then tell monit to restart the process if the condition occurs, as well as adding a condition to get an alert. On macOS, you need the launchctl command to handle the restart. While this is the preferred and 'correct' way to do it, launchctl is a bit of a pain to work with (it seems to assume nobody ever wants to manually restart services), so it's not the approach I would recommend.

  2. Write a script that will trigger the restart, and then specify the path to that script as an exec action for the condition. exec actions implicitly send alerts in current versions of monit, so you don't need to specify any alert condition in this case. In your case, the 'script' can just be the following for the first service:

    #!/bin/bash
/usr/bin/pkill -9 soaagent

    And then something similar for the second one. The only thing it needs to do is somehow trigger launchd restarting the service, it doesn't need to restart it itself. In essence, this approach is just telling monit to trigger the restart just like you are trying to do above, it just does so in a slightly cleaner way that gets the result you want.

Austin Hemmelgarn
  • 11,401
  • 1
  • 24
  • 42
  • So is the issue that including `stop` tells monit to stop monitoring? And why do I need to also launch it in my script. Can't my script just be the single line: `pkill -9 soagent`? The OS will relaunch it right away. – orome Jun 01 '18 at 22:41
  • @orome I've update the answer to hopefully address your comment. – Austin Hemmelgarn Jun 03 '18 at 23:34
  • It's simpler than all this, isn't it: as in my comment. All I need to do is kill the process without letting monit that I've killed it. That way monit will continue to monitor it when macOS restarts it automatically. No? – orome Jun 05 '18 at 12:43
  • @orome But the only ways to do that with monit are what I've listed above. You either define start and stop programs and tell monit to restart it, or you write a script that handles the restart and have monit use that. – Austin Hemmelgarn Jun 05 '18 at 14:03
  • The question is simply "How do I configure to 'monit' monitor a process after it restarts"? You've answered that by just pointing out that I should run a script that kills the process (rather than having monit `stop`) it. – orome Jun 05 '18 at 18:29
  • @orome Because unless you're using something outside monit to kill it and trigger the restart, _that's the only way to do it_. _Something_ has to restart it, and if it's not anything else it has to be monit, in which case the methods I posted are the only options. – Austin Hemmelgarn Jun 05 '18 at 19:48
  • Yes, the fact that it restarts on its own is the whole focus of the question. You’ve given a very complete answer to a different question (from which the answer to my question can be extracted). – orome Jun 05 '18 at 20:42