System: Optimising your Fail2Ban filters

Tweet 0 Shares 0 Tweets 7 Comments

Going beyond the basics with Fail2Ban involves some experience with parsing log files and regular expressions. Below you can find a short introduction to the available tools and steps for analyzing existing filters on your server.

Examining a jail

Let's start with the ssh/sshd filter as it is the only one enabled by default after installation.

In Fail2Ban 0.8.x you can find the settings in /etc/fail2ban/jail.conf:

[ssh]

enabled  = true
port     = ssh
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 5

Whereas in Fail2Ban 0.9.x you need to look in both /etc/fail2ban/jail.conf:

[sshd]

port    = ssh
logpath = %(sshd_log)s

and /etc/fail2ban/jail.d/defaults-*.conf for where it's turned on and where we can override any of the default settings: [sshd] enabled = true maxretry = 3

In Fail2Ban 0.9.x the jail heading in square brackets also identifies the filter being used./p>

In both cases, the jail settings point us to the same filter /etc/fail2ban/filter.d/sshd.conf where you will find definitions for failregex and ignoreregex for this jail. And %(sshd_log)s is mapped to %(syslog_authpriv)s which in turn points to (on Debian) /var/log/auth.log.

What's important is that we've identified both the log file location as well as the filter holding the regular expressions for parsing the log file.

An alternative way to see the filter name and log path is from the command line:

# fail2ban-client status
Status
|- Number of jail:  5
`- Jail list: apache-auth, apache-noscript, apache-overflows, sshd, sshd-ddos

# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed: 0
|  `- File list:  /var/log/auth.log
`- Actions
   |- Currently banned: 0
   |- Total banned: 0
   `- Banned IP list:

How to test the filter

Fail2Ban comes with some handy command line tools. The fail2ban-client interface is useful for querying and managing jails, but in this case the one we want is fail2ban-regex which can be called as follows:

# fail2ban-regex <logfile> <failregex> <ignoreregex>

But instead of typing regular expressions into the command, you can just throw at it the relevant filter configuration file. In other words, to test the sshd filter the command will be:

# fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

Or, if you want to include an ignoreregex definition, include the filter file twice:

# fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf /etc/fail2ban/filter.d/sshd.conf

There some extra options you can include for a better oversight, such as:

-v, --verbose: Be verbose in output
--print-all-missed: Print all missed lines
--print-all-ignored: Print all ignored lines
--print-all-matched: Print all matched lines

Ignoring harmless log lines

Before tweaking our failregex settings we first set up some ignoreregex patterns to filter out what is normal server activity.

For example, on our servers programs such as Systemd, CRON and PostgreSQL can all generate a lot of auth.log activity throughout the day, and to filter it we have added the following expressions:

ignoreregex = : pam_unix\((cron|sshd|systemd-user):session\): session (open|clos)ed for user (daemon|munin|postgres|root)( by \(uid=0\))?$
              : Successful su for (postgres) by root$
              New session \d+ of user (postgres)\.$
              Removed session \d+\.$

For security all regular expressions should be anchored to the end of line $ if at all possible.

Tweaking the filter

With all the background noise filtered out we can now focus on what is being blocked. # fail2ban-regex -v --print-all-missed /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf /etc/fail2ban/filter.d/sshd.conf | less

This will output:

All failregex expressions and the number of matches
All ignoreregex expressions and the number of matches
Date formats detected
Missed lines (lines neither matched nor ignored)

What we're interested in is the last section "Missed lines". Anything listed here has fallen through the cracks and may need our attention. You should consider carefully these lines and modify your settings as necessary to have them included in either the 'matched' or 'ignored' list.

After modifying the filter configuration you can update an already running instance of Fail2Ban using:

# fail2ban-client reload sshd

In our case there's not much to see in auth.log as we secure our SSH by using non-standard ports, certificate based authentication, and other such devices.

Things are much more interesting when you get to http and email filters.

Analyzing an Apache filter

Following the above example, we can now examine other filters using the same approach. Here we're looking into apache-noscript which monitors the Apache error.log looking for spurious requests.

But first, to enable this filter in Fail2Ban 0.8.x you need to edit /etc/fail2ban/jail.conf:

[apache-noscript]

enabled  = true
port     = http,https
filter   = apache-noscript
logpath  = /var/log/apache2/error.log
maxretry = 6
bantime  = 43200

and in Fail2Ban 0.9.x you can put the settings in /etc/fail2ban/jail.d/local.conf or similar. A separate conf file in that directory of every jail is also an option. You only need to add lines that are changed from the default:

[apache-noscript]

enabled  = true
maxretry = 6
bantime  = 43200

Once the jail has been activated and running for some time, you can analyze the performance using:

# fail2ban-regex -v --print-all-missed /var/log/apache2/error.log /etc/fail2ban/filter.d/apache-noscript.conf /etc/fail2ban/filter.d/apache-noscript.conf | less

Looking at the "Missed lines" as before, we've come up with the following regular expressions:

failregex = ^%(_apache_error_client)s ((AH001(28|30): )?File does not exist|(AH01264: )?script not found or unable to stat): /\S*\.(php|aspx?|exe|pl|cgi|rar|zip|gz)(, referer: \S+)?\s*$
            ^%(_apache_error_client)s script '/\S*\.(php|aspx?|exe|pl)\S*' not found or unable to stat(, referer: \S+)?\s*$
            ^%(_apache_error_client)s (script not found or unable to stat|attempt to invoke directory as script): /usr/lib/cgi-bin

ignoreregex = [[]pagespeed:(warn|error)[]]
              [[]client \S+[]] PHP (Notice|Parse error|Warning): 
              [[]client \S+[]] script '.*\.html' not found or unable to stat(, referer: \S+)?\s*$
              [[]client \S+[]] Operation timed out after \d+ milliseconds

These will obviously differ between servers and operating systems. The remaining missed lines mostly relate to IPv6 (support should be coming in a future version of Fail2Ban), or will be captured by other filters such as apache-overflows or apache-wordpress.

Sendmail filters

If you're running a mail server you definitely want to enable one or more of the sendmail jails. The following relates to the sendmail jail in Fail2Ban 0.8.x, enabled in /etc/fail2ban/jail.local:

[sendmail]

enabled  = true
port     = smtp,submission
filter   = sendmail
logpath  = /var/log/mail.log
maxretry = 3

The command for testing the filter is:

# fail2ban-regex --print-all-missed /var/log/mail.log /etc/fail2ban/filter.d/sendmail.conf /etc/fail2ban/filter.d/sendmail.conf | less

And after some work our filters look something like this:

failregex = lost input channel from .*\[<HOST>\] to MTA-v\d after (data|mail|rcpt)$
            \[<HOST>\] did not issue MAIL/EXPN/VRFY/ETRN during connection to (MSP|MTA)-v\d$
            \[<HOST>\], reject.*\.\.\. (Relaying denied)
            \[<HOST>\]: Possible SMTP RCPT flood, throttling\.$
            timeout waiting for input from \[<HOST>\] during server cmd read
            rejecting commands from( .+)? \[<HOST>\] due to pre-greeting traffic
            relay=([^ ]+ )?\[<HOST>\], .* Domain of sender address [\w@.-]+ does not (exist|resolve)$

ignoreregex = sm-mta\[\d+\]: \w+: (from|to)=
              sm-mta\[\d+\]: \S+[[]\d[]]:
              sm-mta\[\d+\]: STARTTLS=(client|server)
              sm-mta\[\d+\]: STARTTLS: (read|write) error=(generic|syscall|timeout)
              : timeout waiting for input from [\w.-]+ during server cmd read$
              : collect: premature EOM: (unexpected close|Connection timed out with [\[\]\w.-]+|Connection reset by \S+)(, sender=\S+)?$
              : collect: (I/O error|read timeout|unexpected close) on connection from
              [\w.-]+ did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-v\d$

As you can see we're quite strict on accepting mail, and any unusual activity will result in a Fail2Ban block - even if Sendmail itself or one of the DNSBL we subscribe to is already addressing the issue.

And any sequence of sendmail blocks will trigger a much longer block triggered by a separate, introspective, Fail2Ban jail.

Ongoing maintenance

You should follow the above steps for all active jails, as well as for any inactive jails that look promising for your installation. And repeat on a regular basis, or at least following any major upgrade.

Don't be afraid to create your own filters based on the ones supplied with the default installation. The command-line tools allow you to test them thoroughly before deployment, and in addition to the simple 'block' action they can also be used to generate emails or run other commands in response to log activity.

Last but not least, when you accidentally manage to block yourself, you can remove the block using:

# fail2ban-client set <jail> unbanip <ip>

System Monitoring the fail2ban log
System Optimising your Fail2Ban filters
System Fail2Ban 0.8.3 Howto
System Implementing Port Knocking with knockd
System Using a Fail2Ban Jail to Whitelist a User
System Blocking FTP Hacking Attempts
System fail2ban and sendmail
System Using systemd to bind fail2ban to nftables
System fail2ban and iptables

< System

User Comments

Post your comment or question

Noah 20 April, 2022

Brilliant post. In particular, your reply to frank solved my issue. This snippet should be used by everyone employing fail2ban for ssh!

[sshd] enabled = false [sshd-aggressive] enabled = true filter = sshd[mode=aggressive]

frank 5 October, 2020

hello,

how can i test jails (not filters)?

i want to ban the messages "Did not receive identification string", so i added to jail.local the "aggressive" lines:

[sshd] enabled = true filter = sshd[mode=aggressive] mode = aggressive

but that didn't work.

Try adding the following to your jail.local and then restarting Fail2Ban:

[sshd] enabled = false

[sshd-aggressive] enabled = true filter = sshd[mode=aggressive]

Adrian 28 January, 2019

Quick question, running ubuntu server and logs such as:
/var/log/apache2/access.log can change at some time in the day.

So if I wish to increase findtime from the standard 10 minutes to say an hour, how would I change it so that both access.log and access.log.1 are included?

Fail2Ban keeps an internal record of what's happened by polling the log file every second. So it doesn't matter - during normal operation - if a log file is rolled over.

But on a busy system I wouldn't apply filters on the Apache access.log as it can be extremely large. Instead try to create a separate log of just the lines you want to monitor in Fail2Ban - using setenvif and CustomLog for example - or PHP error_log().

Adrian 25 January, 2019

That you this paged helped me a lot with the version 0.9.

The first easy to read and follow section I have found on testing filters out

Reredok 13 March, 2018

Thx a great documentation. Helps a lot to dive deeper

pian 21 March, 2017

Hi,

How to create a failregex with this qmail log :

.CN.115.159.31.215 [21/Mar/2017:08:04:37 +0000] 0.901s 0tx "(no subject)" ERR_AUTH_FAILED(504) 0b "(no sender)" 0r "" - ()

Thanks,
Pian

Karl 20 April, 2016

Thanks for this. I've just updated from 0.8 to 0.9 and this was by far the most helpful documentation I could find for the change in layout.