6

Fail2Ban is using a huge amount of memory on my system (1.2 GB). There are several articles which describe how to reduce it. Below is an example for Debian.

  • append the 1ulimit1 command to /etc/default/fail2ban file.

  • Add (to the file) on last line:

    ulimit -s 256

Unfortunately there is no such file or directory on CentOS 7. How to apply this on my system?

After some tips my Systemd file for Fail2Ban is

[Unit] 
Description=Fail2ban Service 

[Service] 
Type=forking 
ExecStart=/usr/bin/fail2ban-client -x start 
ExecStop=/usr/bin/fail2ban-client stop 
ExecReload=/usr/bin/fail2ban-client reload 
PIDFile=/var/run/fail2ban/fail2ban.pid 
Restart=always 
LimitSTACK=256`

Unfortunately the result is still 1251888 KB.

Il Quadrifoglio
  • 141
  • 2
  • 8

5 Answers5

1

/etc/default

The directory /etc/default is never used on any Red Hat based distros. That's a Debian/Ubuntu-ism. For Centos 7 you can take a look at the packages that were installed that relate to fail2ban like so:

$ rpm -aq|grep fail
fail2ban-server-0.9-9.el7.noarch
fail2ban-sendmail-0.9-9.el7.noarch
fail2ban-firewalld-0.9-9.el7.noarch
fail2ban-systemd-0.9-9.el7.noarch
fail2ban-0.9-9.el7.noarch

Contents of fail2ban-server

The fail2ban-server contains the service file for Systemd.

$ rpm -ql fail2ban-server-0.9-9.el7.noarch | grep systemd
/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.py
/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.pyc
/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.pyo
/usr/lib/systemd/system/fail2ban.service

Systemd service file

The contents of the Systemd service file:

$ more /usr/lib/systemd/system/fail2ban.service
[Unit]
Description=Fail2ban Service
After=syslog.target network.target firewalld.service

[Service]
Type=forking
ExecStart=/usr/bin/fail2ban-client -x start
ExecStop=/usr/bin/fail2ban-client stop
ExecReload=/usr/bin/fail2ban-client reload
PIDFile=/var/run/fail2ban/fail2ban.pid
Restart=always

[Install]
WantedBy=multi-user.target

So one could add the extra options to this file, as a quick and dirty way to confirm if they're working.

Long term fixes

To make them permanent, I'd add the options in a more "official" way so that updates to the fail2ban package do not overwrite the modifications to this file. This can be accomplished by adding a customized version of the fail2ban.service file in this directory:

/etc/systemd/system/fail2ban.service

NOTE: A file in this directory, /etc/systemd/system always overrides the default .service file.

However doing it this way have caveats, one being that if a service file is present here when fail2ban were to be updated via yum it would cause the service to be disabled, until you manually reenabled it. So instead you can override fragments of the .service file by adding them to this directory under /etc instead.

Excerpt

To edit a unit file provided by a package, you can create a directory called /etc/systemd/system/unit.d/ for example /etc/systemd/system/httpd.service.d/ and place *.conf files in there to override or add new options. systemd will parse these *.conf files and apply them on top of the original unit. For example, if you simply want to add an additional dependency to a unit, you may create the following file: /etc/systemd/system/unit.d/customdependency.conf

   [Unit]
   Requires=new dependency
   After=new dependency

As another example, in order to replace the ExecStart directive for a unit that is not of type oneshot, create the following file: /etc/systemd/system/unit.d/customexec.conf

   [Service]
   ExecStart=
   ExecStart=new command

so you could create a directory, /etc/systemd/system/fail2ban.service.d and add *.conf files in it with contents like this:

[Service]
ExecStart=
ExecStart=new command

Adding your options there.

Ulimits & Systemd

If you're trying to set a ulimit option for a particular service, then have a look at the man page for systemd.exec.

Excerpt 
LimitCPU=, LimitFSIZE=, LimitDATA=, LimitSTACK=, LimitCORE=, LimitRSS=, 
LimitNOFILE=, LimitAS=, LimitNPROC=, LimitMEMLOCK=, LimitLOCKS=,
LimitSIGPENDING=, LimitMSGQUEUE=, LimitNICE=, LimitRTPRIO=, LimitRTTIME=
These settings control various resource limits for executed processes. See 
setrlimit(2) for details. Use the string infinity to configure no limit 
on a specific resource.

So simply adding LimitSTACK=256 to the customized .conf file that I describe above should give you the same effect as setting ulimit -s 256.

Excerpt - setrlimit(2) man page

If you have a look through the setrlimit(2) man page you can see how the ulimit switches line up with the Systemd limits.

   RLIMIT_STACK
        The maximum size of the process stack, in bytes.  Upon reaching 
        this limit, a SIGSEGV signal is generated.  To handle this signal, 
        a process must employ an alternate signal stack (sigaltstack(2)).

        Since Linux 2.6.23, this limit also determines the amount of space 
        used for the process's  command-line  arguments  and  environment
        variables; for details, see execve(2).

References

slm
  • 363,520
  • 117
  • 767
  • 871
  • So basically I should make the directory fail2ban.service.d and add a *.conf file in it? But it's not clear which code I should use for the file. Please note that I build Fail2Ban manually from the source code. – Il Quadrifoglio Nov 30 '14 at 20:04
  • @user79801 - Yes. Did you look at the referenced link I included? That shows full details, I tried to summarize them here. The last stanza that shows `[service]` would be the way to override the `ExecStart=` with your own version. – slm Nov 30 '14 at 20:46
  • Ok thanx! But what should I use for the ExecStart=? – Il Quadrifoglio Nov 30 '14 at 21:38
  • @user79801 - The method shown by vinc17 should now make more sense given what I've shown above 8-) `ExecStart=/bin/sh -c 'ulimit -s 256; /usr/bin/fail2ban-client -x start'`. Also can you please add a reference to the site where you found that `ulimit` fix? – slm Nov 30 '14 at 22:00
  • Where should I use the code in my init file? I've added 'ulimit -s 256' without any result. – Il Quadrifoglio Dec 01 '14 at 10:20
  • This is my source for the suggested fix. http://hacksnsnacks.com/snippets/reduce-fail2ban-memory-usage/ – Il Quadrifoglio Dec 01 '14 at 10:28
  • @user79801 - see updates. – slm Dec 01 '14 at 13:13
  • After adding the code the result is 1251888 KB. So it doesn't make any change. This is my systemd file [Unit] Description=Fail2ban Service [Service] Type=forking ExecStart=/usr/bin/fail2ban-client -x start ExecStop=/usr/bin/fail2ban-client stop ExecReload=/usr/bin/fail2ban-client reload PIDFile=/var/run/fail2ban/fail2ban.pid Restart=always LimitSTACK=256 – Il Quadrifoglio Dec 01 '14 at 13:40
  • @user79801 - please add that to your A, I can't read it in the comments. – slm Dec 01 '14 at 13:45
  • 2
    I might investigate why your fail2ban is taking up so much memory, and possibly adjust the configuration. On my system, it's more like 9M! – mattdm Dec 01 '14 at 15:15
  • What's the best way to investigate this? my Virtual memory usage for `fail2ban-server` is at `1054644` on an `iRedMail` installation with Resource mem now at ~33600 down from ~96000 after making recommended adjustments to a new file /etc/systemd/system/fail2ban.service.d/memlimit.conf by adding the following only `LimitSTACK=265` to a new `[Service]` header. – Brian Thomas Apr 15 '17 at 20:53
  • 1
    So ill note a couple issues i had here, i was able to reduce it a bit, its still high, but i wanted to note for some reason the accent char at the end, found in the OP, e.g. LimitSTACK=128` or i was getting failures. Also, users need to `systemctl daemon-reload` each time they make these adjustments. Fail2ban is now no longer at the `top` of the list. Another question would be how do we get the virtual mem down? (digital ocean low mem machine). – Brian Thomas Apr 15 '17 at 21:12
0

If your machine has usual sysvinit scripts, you can do that in /etc/init.d/fail2ban (early enough, i.e. before the daemon is started).

If your machine uses systemd, you can do that via fail2ban.service. For instance, instead of

ExecStart=/usr/bin/fail2ban-client -x start

do

ExecStart=/bin/sh -c 'ulimit -s 256; /usr/bin/fail2ban-client -x start'
vinc17
  • 11,912
  • 38
  • 45
  • I've added it to /etc/init.d/fail2ban After that I've done systemctl daemon-reload (system was asking it?) Now the service has been restarted with systemctl fail2ban.service restart. Should everything be ok now? Should I see the changes immediately? – Il Quadrifoglio Nov 30 '14 at 13:25
  • @user79801 If you used `systemctl`, then you're using systemd. So, you probably need the method via `fail2ban.service`. I'm not sure whether you'll see the effect immediately; this is really specific to fail2ban. – vinc17 Nov 30 '14 at 13:51
  • Even if the OP didn't understand it at least this was a pragmatic solution that actually tried to find something working for him! – Florian Heigl May 29 '17 at 00:21
0

The solution was to edit /etc/init.d/fail2ban.

This is the start script:

start() {
echo -n $"Starting fail2ban: "
ulimit -s 256
${FAIL2BAN} -x start > /dev/null
RETVAL=$?
if [ $RETVAL = 0 ]; then
touch ${lockfile}
echo_success
else
echo_failure
fi
echo
return $RETVAL
}

Unfortunately it's only saving me 50 mb

Il Quadrifoglio
  • 141
  • 2
  • 8
0

ulimit affects children processes as well, including notification emails programs like sendmail which may not tolerate such a stacksize limit. This is the case of exim4 sendmail which,,using the proposed memory tweak will segfault instead of emailing like it should when the "recidive" jail is enabled.

raf
  • 1
0

What I usually do is configure something like this in fail2ban.local

[Definition]
dbfile =
dbpurgeage = 0

This forces fail2ban to create a new sqlite database (in memory?) on each restart because what happens is that the persistent database contains all the original log lines that were processed and this usually explodes when the machine is under attack to hundered of MB of useless logs in the sqlite database.

This dbfile originally pointing to /var/lib/fail2ban/fail2ban.sqlite3 is a persistent one so restarting fail2ban needlessly reloads these hundreds of MB of logs from the db (without processing) just to figure out a handful of distinct IPs from some other tables -- go figure!

wqw
  • 101
  • 1