How to start HAProxy on CentOS with systemd

Question

I'm trying to set up a simple web load balancing using CentOS 7.5 (64bit, kernel 3.10), HAProxy 1.8.13 and systemd. It seems that the HAProxy configuration is OK, but starting the application gives me a headache. I tried it with init.d once but was then pointed into the systemd direction, where I'm currently stuck. Been looking for the cause for almost 2 days now and I have reached my limits when it comes to linux, I also couldn't find a cause for this particular behavior anywhere. Most cases like this have some issue in the haproxy config, but that's not the case here apparently.

HAProxy config file validation output

haproxy -f /etc/haproxy/haproxy.conf -c

Configuration file is valid

When I try to start it using systemd, I get the following output in the journalctl:

journalctl -u haproxy.service

Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:50 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:50 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:50 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:51 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:51 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:51 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:51 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:51 localhost.localdomain systemd[1]: start request repeated too quickly for haproxy.service
Aug 13 17:07:51 localhost.localdomain systemd[1]: Failed to start HAProxy Load Balancer.
Aug 13 17:07:51 localhost.localdomain systemd[1]: Unit haproxy.service entered failed state.
Aug 13 17:07:51 localhost.localdomain systemd[1]: haproxy.service failed.

Systemctl shows the following status:

systemctl status haproxy.service

[root@localhost sbin]# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Mo 2018-08-13 17:33:46 CEST; 24min ago
  Process: 1557 ExecStart=/usr/sbin/haproxy -W -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid (code=exited, status=0/SUCCESS)
  Process: 1556 ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q (code=exited, status=0/SUCCESS)
 Main PID: 1557 (code=exited, status=0/SUCCESS)

Aug 13 17:33:46 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:33:46 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:33:46 localhost.localdomain systemd[1]: start request repeated too quickly for haproxy.service
Aug 13 17:33:46 localhost.localdomain systemd[1]: Failed to start HAProxy Load Balancer.
Aug 13 17:33:46 localhost.localdomain systemd[1]: Unit haproxy.service entered failed state.
Aug 13 17:33:46 localhost.localdomain systemd[1]: haproxy.service failed.

The systemd file for this service was mostly taken over from the sample file haproxy 1.8.13 provided. I just replaced some placeholders/variables with fixed values, since some component couldn't handle relative paths.

nano /lib/systemd/system/haproxy.service

[Unit]
Description=HAProxy Load Balancer
After=network.target

[Service]
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/var/run/haproxy.pid"
ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/sbin/haproxy -W -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
ExecReload=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
SuccessExitStatus=143
Type=notify

# The following lines leverage SystemD's sandboxing options to provide
# defense in depth protection at the expense of restricting some flexibility
# in your setup (e.g. placement of your configuration files) or possibly
# reduced performance. See systemd.service(5) and systemd.exec(5) for further
# information.

# NoNewPrivileges=true
# ProtectHome=true
# If you want to use 'ProtectSystem=strict' you should whitelist the PIDFILE,
# any state files and any other files written using 'ReadWritePaths' or
# 'RuntimeDirectory'.
# ProtectSystem=true
# ProtectKernelTunables=true
# ProtectKernelModules=true
# ProtectControlGroups=true
# If your SystemD version supports them, you can add: @reboot, @swap, @sync
# SystemCallFilter=~@cpu-emulation @keyring @module @obsolete @raw-io

[Install]
WantedBy=multi-user.target

It appears that it starts the application, stops it again and then tries to start it again, until it gives up at some point.

When I try to start the application manually from the bash, it looks like nothing is happening:

[root@localhost sbin]# /usr/sbin/haproxy -W -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
[root@localhost sbin]#

No process visible via ps or top. I tried getting some logs, but I couldn't get any log from rsyslog since it doesn't want to write into the file I specified.

Adding the haproxy config file as an appendix, but as per haproxy itself, it seems to be fine:

# Rev1
# HAProxy Global and Default definitions
global
    log 127.0.0.1 local2
    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    maxconn 4000
    user haproxy
    group haproxy
    daemon
        stats socket /var/run/haproxy.sock
        stats timeout 2m

defaults
    log                     global
    option                  dontlognull
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          300s
    timeout server          300s
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

frontend LB_https_vs
    bind *:443
    mode tcp
    option tcplog
    default_backend SP_https_group

backend LB_https_group
    mode tcp
    balance source
    option httpchk GET /MyPage/Logon.aspx
    hash-type consistent
    stick-table type ip size 200k expire 2000s
    stick on src table SP_https_group
    default-server inter 15000
    #server srv1 192.168.1.1:443 id 1 port 443 check
    server srv1 192.168.1.1:443 id 1 check
    server srv2 192.168.1.2:443 id 2 check
    server srv3 192.168.1.3:443 id 3 check
    server srv4 192.168.1.4:443 id 4 check
    server srv5 192.168.1.5:443 id 5 check

listen stats
    mode http
    bind :9000
    stats enable
    stats hide-version
    stats realm HAproxy-Statistics
    stats uri /haproxy_stats
    stats auth admin:default

Any ideas what's going on here?

Thanks in advance

score 2 · Answer 1 · answered Aug 13 '18 at 20:28

2

You need to start haproxy in a mode that runs in foreground. Furthermore, if you want to use Type=notify, you need haproxy to implement the systemd notify daemon.

It looks like you should start it using the -Ws option rather than just -W. From the --help output in the source code:

#if defined(USE_SYSTEMD)
        "        -Ws master-worker mode with systemd notify support.\n"
#endif

Please also note that this only works if haproxy was built with systemd support. Not sure if that's the case for you, but you can check it by looking at the --help output (it doesn't seem to support --help directly, but it prints usage info on unknown args, so --help should work for you.)

You might want to take a look at the suggested haproxy.service template in version 1.8 and the latest one in GitHub, looks like what you're using is close to the latest one, but with some differences (such as -W instead of -Ws...)

answered Aug 13 '18 at 20:28

filbranden

21,113
3
58
84

Thanks for your reply. Apparently I didn't build it with systemd support, this is why I had to switch from -Ws to -W, since it threw an error otherwise. I initially didn't intend to use systemd. If I change the "notify" to something else, could it already work and what downsides would that have? Also I used the haproxy.service.in template from 1.8.13 and only slightly adjusted it (changed -Ws to -W and removed relative paths since I didn't use the makefile to prepare the template). When using --help, I only see the -W parameter, not -Ws. – ArgisIsland Aug 14 '18 at 09:21
I really recommend that you build it with systemd support, that would be best. You will need to install the `systemd-devel` package on your distro, then use `make USE_SYSTEMD=1` to enable support. A close alternative is to use `-W -db` to use worker mode *and* run in foreground, then use Type=simple, that would be closest but it's maybe not exactly the same, there are also some differences in behavior while stopping the daemon, so you'll want to test whether stopping it works as expected. (But, really, just build it with systemd support! You'll be glad you did!) – filbranden Aug 14 '18 at 14:13
1

Thanks for the update. However, in the meantime I tried init.d again and apparently that works now... I'm not exactly sure why though. I used the example init file and renamed the system.d file for haproxy, so it won't be used. I will test your recommendation though so I can check if that fixes the issue with systemd, but I probably won't have the time this week. – ArgisIsland Aug 16 '18 at 11:52

How to start HAProxy on CentOS with systemd

1 Answers1