problems with stunnel systemd unit file

Question

I'd like to create a unit file for stunneland I can't figure out why it's failing.

My stunnel.conf is this:

#Provide the full path to your certificate-key pair file

cert = /etc/pki/tls/certs/stunnel.pem

#lock the process into a chroot jail

chroot = /var/run/stunnel

# and create the PID file in this jail

pid = /stunnel.pid

#change the UID and GID of the process for security reasons
setuid = nobody
setgid = nobody

#enable client mode
client = yes


socket = l:TCP_NODELAY=1
#socket = r:TCP:NODELAY=1

[mysqls]
accept = 127.0.0.1:3306
connect = 10.0.0.3:3307

When I execute stunnel /etc/stunnel/stunnel.conf then it works.

Here's my systemd unit file for stunnel:

[Unit]
;Description=SSL tunnel for network daemons
;Documentation=man:stunnel https://www.stunnel.org/docs.html
After=network.target
After=syslog.target

[Install]
WantedBy=multi-user.target
Alias=stunnel.target

[Service]
Type=forking
User=nobody
Group=nobody
RuntimeDirectory=stunnel
ExecStartPre=-/usr/bin/mkdir /var/run/stunnel
ExecStartPre=-/user/bin/chown nobody:nobody /var/run/stunnel
ExecStart=/bin/stunnel /etc/stunnel/stunnel.conf
ExecStop=/bin/killall -9 stunnel

When I try to start it systemctl start mystunnel.service it fails with

Job for mystunnel.service failed because the control process exited with error code. See "systemctl status mystunnel.service" and "journalctl -xe" for details.

Executing journalctl -xe:

Feb 20 19:26:07 otrs1 polkitd[610]: Registered Authentication Agent for unix-process:14179:2643087 (system bus name :1.62 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, loc
Feb 20 19:26:07 otrs1 systemd[1]: Starting mystunnel.service...
-- Subject: Unit stunnel-otrs.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mystunnel.service has begun starting up.
Feb 20 19:26:07 otrs1 mkdir[14185]: /usr/bin/mkdir: cannot create directory ‘/var/run/stunnel’: File exists
Feb 20 19:26:07 otrs1 systemd[14186]: Failed at step EXEC spawning /user/bin/chown: No such file or directory
-- Subject: Process /user/bin/chown could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /user/bin/chown could not be executed and failed.
--
-- The error number returned by this process is 2.
Feb 20 19:26:08 otrs1 stunnel[14188]: Clients allowed=500
Feb 20 19:26:08 otrs1 stunnel[14188]: stunnel 4.56 on x86_64-redhat-linux-gnu platform
Feb 20 19:26:08 otrs1 stunnel[14188]: Compiled/running with OpenSSL 1.0.1e-fips 11 Feb 2013
Feb 20 19:26:08 otrs1 stunnel[14188]: Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP,FIPS Auth:LIBWRAP
Feb 20 19:26:08 otrs1 stunnel[14188]: Reading configuration from file /etc/stunnel/stunnel.conf
Feb 20 19:26:08 otrs1 stunnel[14188]: FIPS mode is enabled
Feb 20 19:26:08 otrs1 stunnel[14188]: Compression not enabled
Feb 20 19:26:08 otrs1 stunnel[14188]: PRNG seeded successfully
Feb 20 19:26:08 otrs1 stunnel[14188]: Initializing service [mysqls]
Feb 20 19:26:08 otrs1 stunnel[14188]: Certificate: /etc/pki/tls/certs/stunnel.pem
Feb 20 19:26:08 otrs1 stunnel[14188]: Error reading certificate file: /etc/pki/tls/certs/stunnel.pem
Feb 20 19:26:08 otrs1 stunnel[14188]: error queue: 140DC002: error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib
Feb 20 19:26:08 otrs1 stunnel[14188]: error queue: 20074002: error:20074002:BIO routines:FILE_CTRL:system lib
Feb 20 19:26:08 otrs1 stunnel[14188]: SSL_CTX_use_certificate_chain_file: 200100D: error:0200100D:system library:fopen:Permission denied
Feb 20 19:26:08 otrs1 stunnel[14188]: Service [mysqls]: Failed to initialize SSL context
Feb 20 19:26:08 otrs1 stunnel[14188]: str_stats: 12 block(s), 1050 data byte(s), 696 control byte(s)
Feb 20 19:26:08 otrs1 polkitd[610]: Unregistered Authentication Agent for unix-process:14179:2643087 (system bus name :1.62, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Feb 20 19:26:08 otrs1 systemd[1]: stunnel-otrs.service: control process exited, code=exited status=1
Feb 20 19:26:08 otrs1 systemd[1]: Failed to start stunnel-otrs.service.
-- Subject: Unit stunnel-otrs.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit stunnel-otrs.service has failed.
--
-- The result is failed.
Feb 20 19:26:08 otrs1 systemd[1]: Unit mystunnel.service entered failed state.
Feb 20 19:26:08 otrs1 systemd[1]: mystunnel.service failed.

I don't why it gives the error that it can't create the directory (apparently because it exists) but it is not ! Also, why is it failing to read the certificate ? Why running manually it works. SELinux is disabled.

EDIT:

df -h

Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/cl-root   14G  1.9G   13G  14% /
devtmpfs             234M     0  234M   0% /dev
tmpfs                245M   54M  191M  22% /dev/shm
tmpfs                245M  4.4M  240M   2% /run
tmpfs                245M     0  245M   0% /sys/fs/cgroup
/dev/sda1           1014M  138M  877M  14% /boot
tmpfs                 49M     0   49M   0% /run/user/0

EDIT2:

After applying suggestions from ErikF the directory exists problem went away but it's still failing on reading the certificate:

   Feb 20 20:42:59 otrs1 polkitd[610]: Registered Authentication Agent for unix-process:16232:3104221 (system bus name :1.73 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, loc
    Feb 20 20:42:59 otrs1 systemd[1]: Starting stunnel-otrs.service...
    -- Subject: Unit stunnel-otrs.service has begun start-up
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit stunnel-otrs.service has begun starting up.
    Feb 20 20:42:59 otrs1 stunnel[16239]: Clients allowed=500
    Feb 20 20:42:59 otrs1 stunnel[16239]: stunnel 4.56 on x86_64-redhat-linux-gnu platform
    Feb 20 20:42:59 otrs1 stunnel[16239]: Compiled/running with OpenSSL 1.0.1e-fips 11 Feb 2013
    Feb 20 20:42:59 otrs1 stunnel[16239]: Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP,FIPS Auth:LIBWRAP
    Feb 20 20:42:59 otrs1 stunnel[16239]: Reading configuration from file /etc/stunnel/stunnel.conf
    Feb 20 20:42:59 otrs1 stunnel[16239]: FIPS mode is enabled
    Feb 20 20:42:59 otrs1 stunnel[16239]: Compression not enabled
    Feb 20 20:42:59 otrs1 stunnel[16239]: PRNG seeded successfully
    Feb 20 20:42:59 otrs1 stunnel[16239]: Initializing service [mysqls]
    Feb 20 20:42:59 otrs1 stunnel[16239]: Certificate: /etc/pki/tls/certs/stunnel.pem
    Feb 20 20:42:59 otrs1 stunnel[16239]: Error reading certificate file: /etc/pki/tls/certs/stunnel.pem
    Feb 20 20:42:59 otrs1 stunnel[16239]: error queue: 140DC002: error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib
    Feb 20 20:42:59 otrs1 stunnel[16239]: error queue: 20074002: error:20074002:BIO routines:FILE_CTRL:system lib
    Feb 20 20:42:59 otrs1 stunnel[16239]: SSL_CTX_use_certificate_chain_file: 200100D: error:0200100D:system library:fopen:Permission denied
    Feb 20 20:42:59 otrs1 stunnel[16239]: Service [mysqls]: Failed to initialize SSL context
    Feb 20 20:42:59 otrs1 stunnel[16239]: str_stats: 12 block(s), 1050 data byte(s), 696 control byte(s)
    Feb 20 20:42:59 otrs1 polkitd[610]: Unregistered Authentication Agent for unix-process:16232:3104221 (system bus name :1.73, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
    Feb 20 20:42:59 otrs1 systemd[1]: stunnel-otrs.service: control process exited, code=exited status=1
    Feb 20 20:42:59 otrs1 systemd[1]: Failed to start stunnel-otrs.service.
    -- Subject: Unit stunnel-otrs.service has failed
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit stunnel-otrs.service has failed.
    --
    -- The result is failed.
    Feb 20 20:42:59 otrs1 systemd[1]: Unit stunnel-otrs.service entered failed state.
    Feb 20 20:42:59 otrs1 systemd[1]: stunnel-otrs.service failed.

Any ideas or hints please? 

Appreciate or help!

When you say that it manually works, which commands are you running as `root`, and which commands as `nobody`? — Mark Stosberg, Mar 06 '18 at 21:51
@MarkStosberg when I run it manually I execute the command `stunnel /etc/stunnel/stunnel.conf` as `root`. I want to lock the process into a chroot jail and to create a PID file in this jail and change the UID and GID for security reasons to `nobody`. But chances that I'm confusing something are quit high :) — yesOrMaybeWhatever, Mar 06 '18 at 21:56

score 7 · Answer 1 · edited Aug 13 '21 at 08:51

How to approach this

A service that is to run under a service manager should not attempt to self-daemonize, should not use the rickety and dangerous PID file mechanism, and should not (in the overwhelmingly common case) drop privileges itself. All of that will be done by the service management system, properly.

So when you ran stunnel at the command line like that, it should have made the shell wait until you terminated/stopped the process.

All of the running under the aegis of an unprivileged user account, setting up reduced filesystem access, and indeed (for those programs that are capable of inheriting and using open socket file descriptors) opening the listening socket is the domain of service management.

This is what the systemd people naïvely call the "new style" way of running dæmons. In fact, it is just the way that daemontools users have been saying to run dæmons for the past two decades, and that IBM has been saying for the past quarter of a century.

That unprivileged user account should not be nobody, which can be listed as the owner of files. The service in question does not need ownership permission on any files or directories at all, so the unprivileged user account should be one created expressly for this purpose, with something like:

useradd --shell /usr/bin/true mysql-stunnel-d

A chroot() environment for this is actually non-trivial to set up right in systemd, because /bin/stunnel and all of the files that it reads (which according to its doco in addition to the certificate file and configuration file also includes quite a lot of system stuff; with things like /dev/zero, the NSS configuration system, and the timezone database) need to be configured with BindReadOnlyPaths. A simpler approach is the Protect… mechanism.

Configuration files

The socket is described by a socket unit. stunnel does not understand the LISTEN_FDS protocol, but it is UCSPI-TCP compatible. So the socket unit has to describe an accepting socket:

; /etc/systemd/service/mysql-stunnel.socket
[Unit]
Description=SSL wrapper for MySQL
Documentation=

[Socket]
ListenStream=127.0.0.1:mysql
ListenStream=[::1]:mysql
Accept=yes
NoDelay=yes

[Install]
WantedBy=multi-user.target

Notice that it is the socket that gets enabled/disabled/started/stopped here, with systemctl. The service, described by a service unit, is automatically started on demand by the socket. Because it is an accepting socket, it is a template service unit. It describes all of the privilege dropping and setup that service management does:

; /etc/systemd/service/[email protected]
[Unit]
Description=SSL wrapper for MySQL
Documentation=

[Service]
Type=simple
User=mysql-stunnel-d
ProtectHome=yes
ProtectSystem=strict
PrivateTmp=yes
StandardInput=socket
StandardOutput=socket
StandardError=journal
ExecStart=/bin/stunnel /etc/stunnel/mysql-stunnel.conf

The stunnel configuration file does not then deal in any of the stuff that service management does for it:

# /etc/stunnel/mysql-stunnel.conf
cert = /etc/pki/tls/certs/stunnel.pem
client = yes
foreground = yes
connect = 10.0.0.3:3307

Bonus content

Yes, this is the daemontools way. Pass these two unit files through convert-systemd-units from the nosh toolset and one gets a run and a service program (amongst several others) that exemplify how this is done the daemontools way, with a suitable set of chain-loading UCSPI and other tools:

% system-control convert-systemd-units ./mysql-stunnel.socket
convert-systemd-units: WARNING: ./[email protected]: Forcing setting: [Service] StandardError = log
convert-systemd-units: WARNING: ./mysql-stunnel.socket: Unused setting: [unit] documentation = 
convert-systemd-units: WARNING: ./[email protected]: Unused setting: [service] standarderror = journal
convert-systemd-units: WARNING: ./[email protected]: Unused setting: [unit] documentation = 
% system-control cat ./mysql-stunnel
start:#!/bin/nosh                
start:#Start file generated from ./mysql-stunnel.socket
start:true
stop:#!/bin/nosh
stop:#Stop file generated from ./mysql-stunnel.socket
stop:true
run:#!/bin/nosh
run:#Run file generated from ./mysql-stunnel.socket
run:#SSL wrapper for MySQL
run:tcp-socket-listen 127.0.0.1 mysql
run:tcp-socket-listen "::1" mysql
run:move-to-control-group ../mysql-stunnel.service
run:envuidgid --supplementary -- mysql-stunnel-d
run:userenv-fromenv
run:unshare --mount
run:set-mount-object --recursive slave /
run:make-private-fs --temp --homes
run:make-read-only-fs --os --etc
run:set-mount-object --recursive shared /
run:setuidgid --supplementary -- mysql-stunnel-d
run:tcp-socket-accept --no-delay
run:./service
service:#!/bin/nosh
service:#Service file generated from ./[email protected]
service:#SSL wrapper for MySQL
service:/bin/stunnel /etc/stunnel/mysql-stunnel.conf
restart:#!/bin/sh
restart:#Restart file generated from ./[email protected]
restart:exec false  # ignore script arguments
%

Jonathan de Boyne Pollard (2015). [The gen on the UNIX Client-Server Program Interface] (https://jdebp.uk/FGA/UCSPI.html). Frequently Given Answers.
Jonathan de Boyne Pollard (2018). convert-systemd-units. nosh Guide. Softwares.
Jonathan de Boyne Pollard (2018). Don't abuse nobody for running dæmons.. Frequently Given Answers.
Jonathan de Boyne Pollard (2018). Mistakes to avoid when designing Unix dæmon programs. Frequently Given Answers.

score 1 · Answer 2 · answered Mar 07 '18 at 03:36

1

You have a spelling mistake:

ExecStartPre=-/user/bin/chown nobody:nobody /var/run/stunnel

You probably meant /usr/bin/chown, not /user/bin/chown.

Also, if /var/run is a symlink to /run, you could probably replace the directory-creation commands with RuntimeDirectory=

RuntimeDirectory=stunnel

answered Mar 07 '18 at 03:36

ErikF

3,942
1
10
15

2

You and M. Stosberg have missed a whole bunch of other errors being made here, from [abusing `nobody`](http://jdebp.eu./FGA/dont-abuse-nobody-for-daemons.html), through readiness protocol mismatch and [wrongheaded forking plus use of the rickety and dangerous PID file mechanism](http://jdebp.eu./FGA/unix-daemon-design-mistakes-to-avoid.html), to trying to `chroot()` in an unprivileged program (part of a larger problem of having the service program itself trying to duplicate what the service manager does). And you missed that the service unit is already using `RuntimeDirectory` _and_ `mkdir`. – JdeBP Mar 07 '18 at 06:06
@ErikF Thank you for spotting the typo! And in fact after adding RuntimeDirectory=stunnel the directory already exists went away. But it's still failing on reading the certificate - please see my updated question. – yesOrMaybeWhatever Mar 07 '18 at 07:08
Is `/etc/pki/tls/certs` actually readable by `nobody`? Generally services can read these because they start as a privileged account and then drop privileges after forking, but because you're starting the service as `nobody`, everything has to be readable as `nobody`. Try removing the `User=` and `Group=`: you should see that the forked process will still have restricted rights. – ErikF Mar 07 '18 at 08:36
@ErikF thank you, now it "works" I mean I can start the service and it does what it supposed to do although looking and what JdeBP provided I'm not quit sure how I could possibly correct my mistakes. Sorry that I cannot up vote your comments and answer but apparently I need to get more reputation but your help is much appreciated :) – yesOrMaybeWhatever Mar 07 '18 at 10:02

score 1 · Answer 3 · answered Aug 22 '20 at 15:43

Persisting Stunnel

Since stunnel deals in ssl connections, which implies the use of TCP rather than UDP, it is not unreasonable to assume the connections are meant to be long lived, and for those cases I have found @JdeBP's answer to be absolutely correct; it has become my reference point for the right way to do this kind of unit, in particular with Stunnel.

However, where there is a significant cost to starting a process relative to the lifetime of the process, having systemd sockets listening for the connection can make less sense. For comparison - you would not necessarily want your web server firing up a new instance for each incoming TCP connection, certainly when lightweight threads are an option.

stunnel, similarly, has a few optimisations it can utilise if it is running as a persistent process rather than being started for each connection. Furthermore, you won't be starting a new process for each connection but as I understand it stunnel (depending on the compilation options) can use threads which are much cheaper to instantiate - and you avoid potential disc hits for certain workload distributions.

So how do we run stunnel as a persistent service, whilst heeding all of the excellent advice provided by @JdeBP's answer (Avoiding PID files, not abusing 'nobody', and not forking)?

The following was the base I used for connecting to a service which was protected by ssl, using a client which was not encryption aware. You may have to tweak if your use case is different.

;/etc/stunneld.conf

client=yes 
;presuming that the upstream is ssl protected and the downstream is not

foreground=yes
;no forking. Use `quiet` to get less noisy logs

pid=
;no PID File, note that it must be blank to avoid PID creation

[my-upstream-service]
accept=x.x.x.x:pppp
;optional address, and mandatory port to listen on - can be a hostname instead of an address

connect=x.x.x.x:pppp 
;upstream ssl protected service

CApath=/etc/ssl/certs
;may be different in your distribution

verifyChain=yes
checkHost=example.com
;were required in my case as the ssl cert was specific to a hostname. Note that without the verification you're still vulnerable to MITM attacks.

;/etc/systemd/system/stunneld.service (location may be different in your distro)

[Unit]
Description=Stunnel Service
Documentation=

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
User=stunnel4
;my distro happened to provide a stunnel user as part of the package. YMMV
ProtectHome=yes
ProtectSystem=strict
PrivateTmp=yes
StandardError=journal
ExecStart=/usr/bin/stunnel /etc/stunneld.conf

These configs allow us to run stunnel as a persistent task, which gives us the optimisation benefits associated with that in the context where it is doing many small connections rather than a few large ones.

score 0 · Answer 4 · answered Mar 06 '18 at 21:50

0

As seen in man systemd.service under PermissionStartOnly=, by default ExecStartPre= command will be run as the user provided in User=.

From your example, it appears you intend them to run as root, so you should set:

 PermissionStartOnly=true

answered Mar 06 '18 at 21:50

Mark Stosberg

7,420
1
32
42

Unfortunately it's still failing after adding the line. – yesOrMaybeWhatever Mar 06 '18 at 22:10

score 0 · Answer 5 · answered Nov 29 '19 at 17:26

0

In my case I did not have any configuration file.

For that I have this configuration file:

client = yes

[stunnel]
accept = 127.0.0.1:41194
connect = vpnserverhost:port

placed as /etc/stunnel/stunnel.conf

answered Nov 29 '19 at 17:26

Mahdi-Malv

101
2

problems with stunnel systemd unit file

5 Answers5

How to approach this

Configuration files

Bonus content

Further reading

Persisting Stunnel