/usr/bin/mail call in ExecStart not sending any mail

Question

I have the following service configured:

[Unit]
Description=SCollector
After=NetworkManager.service

[Service]  
Type=simple
ExecStart=/bin/sh -c "/opt/scollector/scollector /opt/scollector/collectors || (echo '' | /usr/bin/mail -s 'scollector died' [email protected] && exit -1)"
Restart=on-failure

[Install]
WantedBy=multi-user.target

For some reason, that mail command never sends any mail when the scollector process exits with non-0. This works AOK when run on the command line, /bin/sh call and all. I've captured STDOUT and STDERR of mail, and it is throwing no errors. There is nothing in maillog.

What gives? Why won't it send mail?

your `ExecStart` runs `exit -1` when `scollector` exits with `0`. Is it ok? — Evgeny Vereshchagin, Sep 22 '15 at 21:31
Yeah, you're correct. I ended up not using any of the above, so I never ran into that `exit`. I'll fix it in the question regardless so it doesn't red-herring anyone. — alienth, Sep 22 '15 at 21:36
try `ExecStart=/bin/sh -c "false || (echo '' | /usr/bin/mail -s 'scollector died' [email protected] && exit -1)"`. Does it work? Which version of `systemd` are you using? — Evgeny Vereshchagin, Sep 22 '15 at 22:06
Nope. As indicated in my answer, the issue is caused by `systemd` nuking the double-forked `sendmail` call from `mail`. I'm on cent7, which uses `systemd 208`. — alienth, Sep 22 '15 at 22:08
To reiterate: simply having `ExecStart=/bin/sh -c "/usr/bin/mail -s test [email protected] — alienth, Sep 22 '15 at 22:18
oh, sorry. I missed your answer:) Which version of `sendmail` are you using? works fine with `systemd 219`, `sendmail 8.14.4` — Evgeny Vereshchagin, Sep 22 '15 at 22:47
`mail` is calling `/usr/sbin/sendmail`, but that's just a shim over to `/usr/sbin/sendmail.postfix`. I'm on postfix 2.10. — alienth, Sep 22 '15 at 22:51

alienth · Answer 1 · 2015-09-22T22:33:46.730

/usr/bin/mail performs a double fork to daemonize sendmail for sending the email. This sendmail proc gets reowned to init, so normally it wouldn't be affected by anything that happens with the original parent - except in the systemd case that reowned grandchild is still in the same cgroup as the original service. When systemd tears things down, it kills all processes within the cgroup, including the reowned sendmail process.

The mail command itself ran fine, but sendmail was getting killed by systemd before it had a chance to do its thing.

You can get around this by setting KillMode in the Unit section to process (the default is control-group). That will cause systemd to only kill the process which it directly fired.

Interestingly the way I stumbled upon this was through the use of strace. A normal strace revealed nothing, but the mail suddenly started working when using strace -f. strace -f was causing the main process to stick around until all of the children and orphaned grandchildren had wrapped up.

`-S sendwait` as in `cat /tmp/mailtext | /usr/bin/mailx -S sendwait -r [email protected] -s "My Working Mail" [email protected]` works for me — mnagel, Feb 18 '16 at 14:08

score 2 · Answer 2 · answered Oct 04 '15 at 12:30

The questioner has identified the problem; but xyr solution is a bodge, and xyr description of the mechanics is incorrect.

The mail command does not perform a double fork. It forks just once, and the sendmail shim process is its immediate child that is not reparented to anything. It simply chooses whether to waitpid() for that child or not, before it exits.

The same is true of the sendmail shim itself. It does not double fork. On some MTSes it doesn't even fork at all. On others it forks just the once and chooses whether to wait or not dependent from some configurable "delivery mode" option.

The correct way to get around the problem is twofold:

Set mailx's documented and standardized sendwait option. That specifically addresses the problems of asynchronous enqueueing, by making mailx wait for the sendmail shim child process to finish. (Sadly, even though this option has been around since at least 1986 and is documented for mailx in the SVID, bsd-mailx does not have it. heirloom-mailx has it, though.)
Set whatever MTS is in use to use a synchronous queueing/delivery mode if it isn't using one already.
- If using netqmail, do nothing. netqmail's sendmail shim is always queued and synchronous, directly chain loading through qmail-inject to qmail-queue without forking at all.
- If using Postfix, do nothing. Postfix's sendmail shim is always queued and synchronous, forking once and waiting for postdrop to finish before exiting itself.
- exim has the -odf command line option.

/usr/bin/mail call in ExecStart not sending any mail

2 Answers2

Further reading

Linked