Dynamic IP Denylisting with NGINX Plus and fail2ban

You may not realize it, but your website is under constant threat. If it’s running WordPress, bots are trying to spam you. If it has a login page, there are brute‑force password attacks. You may also consider search engine spiders as unwanted visitors.

Defending your site from unwanted, suspicious, and malicious activity is no easy task. Web application firewalls, such as NGINX App Protect and the NGINX Plus Certified Modules available from our partners, are effective tools and must be considered as part of your security stack. For most environments, there’s no such thing as too much security, and a multi‑layered approach is invariably the most effective.

In this blog post, we’ll discuss the use of fail2ban as another layer of the web security stack. Fail2ban is an intrusion detection system (IDS) which continually monitors log files for suspicious activity, and then takes one or more preconfigured actions. Typically fail2ban monitors for failed login attempts and then blocks (bans) the offending IP address for a period of time. This is a simple yet effective defense against brute‑force password attacks.

In NGINX Plus R13 we introduced a native key‑value store and a new NGINX Plus API. This allows for a wealth of dynamic configuration solutions in which NGINX Plus configuration and behavior can be driven by an external system. In this blog post, we’ll also show how fail2ban can be used to automatically reconfigure NGINX Plus to ignore requests from IP addresses that have caused multiple failed authentication events.

Editor – In NGINX Plus R16 and later, key‑value stores can be synchronized across all NGINX Plus instances in a cluster. (State sharing in a cluster is available for other NGINX Plus features as well.) For details, see our blog and the NGINX Plus Admin Guide.

Using a Key‑Value Store for IP Denylisting

The NGINX Plus key‑value store is a native, in‑memory store with three primary characteristics:

1. Key‑value pairs are represented as JSON objects
2. Key‑value pairs are managed entirely through an API
3. The values are available to NGINX Plus as regular configuration variables

The key‑value store is defined by creating a shared memory zone named by the keyval_zone directive. This key‑value store can then be populated with an initial set of values by using the HTTP POST method to submit a JSON object to the API. The keyval directive then defines which existing variable will be used as the lookup key ($remote_addr in the example), and the name of a new variable ($num_failures) that will be evaluated from that key’s corresponding value.

The API is enabled by designating a location block as the NGINX Plus API endpoint.

server {
    listen 1111;
    allow  127.0.0.1; # Only allow access from localhost,
    deny   all;       # and prevent remote access.

    location /api {
        api write=on; # The NGINX Plus API endpoint in read/write mode
    }
}
keyval_zone zone=denylist:1M;
keyval $remote_addr $num_failures zone=denylist;

server {
    listen 80;

    location / {
        root /usr/share/nginx/html;
        if ($num_failures) {
            return 403;
        }
    }
}

vim: syntax=nginx

Before we add key‑value pairs to it, a request for the contents of the denylist store returns an empty JSON object.

$ curl http://localhost:1111/api/6/http/keyvals/denylist{}

The key‑value store can now be populated with an initial key‑value pair using the HTTP POST method (in the form of the -d argument to the curl command) to submit a new JSON object. Multiple key‑value pairs can be POST‑ed to an empty key‑value store and singularly thereafter.

$ curl -iX POST -d '{"10.0.0.1":"1"}' http://localhost:1111/api/6/http/keyvals/denylistHTTP/1.1 201 Created
...

A key‑value pair is removed by PATCH‑ing the key with a value of null.

$ curl -iX PATCH -d '{"10.0.0.1":null}' http://localhost:1111/api/6/http/keyvals/denylistHTTP/1.1 204 No Content
...

All key‑value pairs can be removed from the key‑value store by sending the DELETE method.

$ curl -iX DELETE http://localhost:1111/api/6/http/keyvals/denylistHTTP/1.1 204 No Content
...

A simple implementation of IP denylistlisting can now be configured.

server {
    listen 1111;
    allow  127.0.0.1; # Only allow access from localhost,
    deny   all;       # and prevent remote access.

    location /api {
        api write=on; # The NGINX Plus API endpoint in read/write mode
    }
}
keyval_zone zone=denylist:1M;
keyval $remote_addr $num_failures zone=denylist;

server {
    listen 80;

    location / {
        root /usr/share/nginx/html;
        if ($num_failures) {
            return 403;
        }
    }
}

vim: syntax=nginx

This configuration snippet configures port 80 as a web server. Line 18 evaluates the $num_failures variable to the value part of the key‑value pair in the denylist shared memory zone that matches the key that corresponds to the $remote_addr variable (client IP address). This process of evaluating $num_failures is more clearly expressed sequentially:

1. Pass the value of $remote_addr to the denylist key‑value store
2. If $remote_addr is an exact match for a key then obtain the value of that key‑value pair
3. Return the value as $num_failures

And so, if the client IP address has been POST‑ed to the key‑value store then all requests result in an HTTP 403 Forbidden error. The advantage of this approach, as opposed to using a map block for the same purpose, is that the IP denylist can be controlled by an external system.

Using fail2ban to Dynamically Manage the IP Denylist

As mentioned above, fail2ban is commonly used to detect suspicious and/or malicious activity in log files, and then take action to protect the system. The default action is to configure iptables to drop all packets originating from the IP address that was logged. This approach, although effective in blocking a malicious actor, provides a very poor user experience for genuine users, especially if they’ve forgotten their password. To these users, it appears as if the website is simply unavailable.

The following configuration describes a fail2ban action that submits the offending IP address to the NGINX Plus denylist key‑value store. NGINX Plus then displays a more helpful error page and simultaneously applies a request rate limit to that IP address so that if the actions are part of an attack, the attack is effectively neutralized.

We assume a default installation of fail2ban on the same host as NGINX Plus, with all configuration under the /etc/fail2ban directory.

The following fail2ban action uses the NGINX Plus API to add and remove “banned” IP addresses within the denylist key‑value store in the same way as our simple example above. We place the nginx-plus-denylist.conf file in the /etc/fail2ban/action.d directory.

[Definition]
actionban = curl -s -o /dev/null -d '{"":""}' http://localhost:1111/api/6/http/keyvals/denylist
actionunban = curl -s -o /dev/null -X PATCH -d '{"":null}' http://localhost:1111/api/6/http/keyvals/denylist

The following fail2ban jail enables the built‑in filter that detects failed login attempts using NGINX’s HTTP Basic Authentication module and applies the action defined in nginx-plus-denylist.conf. Many other built‑in filters are available, and can be easily created to detect unwanted activity in NGINX access logs.

[DEFAULT]
bantime   = 120
banaction = nginx-plus-denylist
 
[nginx-http-auth]
enabled = true

The following NGINX Plus configuration snippet applies HTTP Basic authentication to the default “Welcome to NGINX” page. We place the password_site.conf file in the /etc/nginx/conf.d directory.

server {
    listen 1111;
    allow  127.0.0.1; # Only allow access from localhost,
    deny   all;       # and prevent remote access.

    location /api {
        api write=on; # The NGINX Plus API endpoint in read/write mode
    }
}

keyval_zone zone=denylist:1M state=denylist.json;
keyval $remote_addr $num_failures zone=denylist;

limit_req_zone $binary_remote_addr zone=20permin:10M rate=20r/m;

server {
    listen 80;
    root /usr/share/nginx/html;

    location / {
        auth_basic "closed site";
        auth_basic_user_file users.htpasswd;

        if ($num_failures) {
            rewrite ^.* /banned.html;
        }
    }

    location = /banned.html {
        limit_req zone=20permin burst=100;
    }
}

vim: syntax=nginx

Line 11 defines the keyval_zone as in denylist_keyval.conf above, but with the addition of the state parameter, which specifies a file where the state of the key‑value store is stored and so persists when NGINX Plus is stopped and restarted. The key‑value store persists across regular configuration reloads without specifying a state file. (Lines 1–10 are not shown, but are the same as in denylist_keyval.conf.)

Line 14 creates a shared memory zone called 20permin and specifies a maximum request rate of 20 requests per minute for each client IP address. This rate limit is then applied when an IP address is added to the denylist by fail2ban (line 30).

The server block defines a web server listening on the default port (80), where all requests are matched by the location / block (line 20). The root directive (line 18) specifies where our web content resides. Lines 21–22 specify that HTTP Basic authentication is required for this site and that the database of passwords for authorized users is in the file users.htpasswd. The documentation for the auth_basic_user_file directive describes the format of that file. Note that this configuration is unencrypted for testing purposes and any production web site using HTTP Basic authentication should use SSL/TLS for transport security.

When the client IP address has been denylisted (lines 24–26), instead of returning the HTTP 403 error code, we rewrite the request to be /banned.html which is then processed in the location block that exactly matches that URI (line 29). This returns a static web page that explains that the user’s IP address has been blocked due to excessive login failures. It also applies a restrictive rate limit (line 30) that prevents a malicious client from unnecessarily consuming system resources.

(The HTML for this sample web page is available as banned.html in the GitHub repository for this post.)

We can simulate successive failed login attempts and the corresponding fail2ban activity as follows.

$ curl -i http://admin:password@www.example.com/HTTP/1.1 401 Unauthorized ...
$ curl -i http://admin:admin@www.example.com/
HTTP/1.1 401 Unauthorized ...
$ curl -i http://admin:pass1234@www.example.com/
HTTP/1.1 401 Unauthorized ...
$ curl -i http://admin:letmein@www.example.com/
HTTP/1.1 401 Unauthorized ...
$ curl -i http://admin:fred@www.example.com/
HTTP/1.1 401 Unauthorized ...
$ curl  http://admin:P@ssw0rd@www.example.com/
<!DOCTYPE html>
<html>
<head>
<title>Banned</title>
...
$ tail -f /var/log/fail2ban.log
2017-09-15 13:55:18,903 fail2ban.filter   [28498]: INFO    [nginx-http-auth] Found 172.16.52.1
2017-09-15 13:55:28,836 fail2ban.filter   [28498]: INFO    [nginx-http-auth] Found 172.16.52.1
2017-09-15 13:57:49,228 fail2ban.filter   [28498]: INFO    [nginx-http-auth] Found 172.16.52.1
2017-09-15 13:57:50,286 fail2ban.filter   [28498]: INFO    [nginx-http-auth] Found 172.16.52.1
2017-09-15 13:57:52,015 fail2ban.filter   [28498]: INFO    [nginx-http-auth] Found 172.16.52.1
2017-09-15 13:57:52,088 fail2ban.actions  [28498]: NOTICE  [nginx-http-auth] Ban 172.16.52.1
2017-09-15 13:59:52,379 fail2ban.actions  [28498]: NOTICE  [nginx-http-auth] Unban 172.16.52.1

This dynamic IP denylisting solution can now run without any further configuration changes. Fail2ban watches the NGINX log files and adds banned IP addresses to the NGINX Plus key‑value store using the API. After 120 seconds (the bantime configured in jail.local), the offending IP address is removed from the denylist, again using the NGINX Plus API, and login attempts are once more accepted from that address.

The NGINX Plus API and key‑value store make this type of integration possible. Advanced configuration solutions can now be achieved without requiring the construction of NGINX Plus configuration files and performing reloads. We’d love to hear about how you’re using these features to create your own dynamic configuration solutions. Please add a comment below.

To try NGINX Plus, start your free 30-day trial today or contact us to discuss your use cases.