Improving NGINX Performance with Kernel TLS and SSL_sendfile( )

Transport Layer Security (TLS) is an extremely popular cryptography protocol. Implementing TLS in the kernel (kTLS) improves performance by significantly reducing the need for copying operations between user space and the kernel.

Combining kTLS and sendfile() means data is encrypted directly in kernel space, before being passed to the network stack for transmission. This eliminates the need to copy data into user space to be encrypted by TLS libraries and then back into kernel space for transmission. kTLS also enables offload of TLS processing to hardware, including offload of TLS symmetric crypto processing to network devices.

Modern Linux and FreeBSD kernels support offloading TLS to the kernel, and now NGINX Open Source does too! NGINX 1.21.4 introduces support for kTLS when serving static files and cached responses with SSL_sendfile(), which can hugely improve performance. As detailed below, both the kernel and OpenSSL must be built with kTLS for NGINX to use SSL_sendfile().

In this blog we detail which operating system and OpenSSL versions support kTLS, and show how to build and configure the kernel and NGINX for kTLS. To give you an idea of the performance improvement you can expect from kTLS, we also share the specs and results of our testing on FreeBSD and Ubuntu.

Note: kTLS implementations are quite new and evolving rapidly. This blog describes support for kTLS as of November 2021, but keep an eye out for announcements on nginx.org and the NGINX blog<.htmla> about changes to the information and instructions provided here.

General Requirements

• Operating system – Either of:

  • FreeBSD 13.0+. As of November 2021, FreeBSD 13.0+ is the only OS that supports kTLS in NGINX without a manual build of NGINX to incorporate OpenSSL 3.0.0+. See Enabling NGINX with kTLS on FreeBSD.

  • A Linux distribution built on Linux kernel version 4.17 or later, though we recommend using those built on version 5.2 or later when possible. (kTLS support is actually available in version 4.13, but OpenSSL 3.0.0 requires kernel header version 4.17 or later.)

• OpenSSL – Version 3.0.0 or later

• NGINX – Version 1.21.4 or later (mainline)

  [Editor – NGINX Plus R27 and later supports kTLS on eligible Linux‑based operating system versions; NGINX Plus R26 and later supports it on eligible FreeBSD versions. For details about supported OSs, see the NGINX Plus Releases page.]

Operating System Support

OSs That Support kTLS

As of November 2021, of the OSs supported by NGINX Open Source, the following support kTLS and the indicated ciphers. For details about cipher support, see TLS Protocol and Cipher Support.

  * Kernel version must be 5.10, not 4.14; see OSs That Do Not Support kTLS and the Amazon Linux 2 FAQ
 ** Inherits its kTLS support status from RHEL 8 as its upstream source
*** See the FreeBSD commit log

OSs That Do Not Support kTLS

The following OSs do not support kTLS, for the indicated reason:

• Alpine Linux 3.11–3.14 – Kernel is built with the CONFIG_TLS=n option, which disables building kTLS as a module or as part of the kernel.
• Amazon Linux 2 – Linux kernel version is 4.14 for the default Amazon Linux 2 AMI (see the Amazon Linux 2 FAQ).
• CentOS 7.4+ – Linux kernel version is 3.10. Inherits its kTLS support status from RHEL 7.4+ as its upstream source.
• Debian 10 and 11 – Kernel is built with the CONFIG_TLS=n option (see the Debian bug report logs).
• RHEL 7.4+ – Linux kernel version is 3.10.
• SLES 12 SP5+ – Linux kernel version is 4.12.
• Ubuntu 18.04 LTS – Linux kernel version is 4.15.

TLS Protocol and Cipher Support

As detailed above, OSs that support kTLS vary in their support for TLS protocols and ciphers.

With TLSv1.2, the kTLS module supports these ciphers:

• AES128-GCM-SHA256
• AES256-GCM-SHA384
• ECDHE-RSA-AES128-GCM-SHA256
• ECDHE-RSA-AES256-GCM-SHA384

With TLSv1.3, the kTLS module supports these cipher suites:

• TLS_AES_128_GCM_SHA256
• TLS_AES_256_GCM_SHA384
• TLS_CHACHA20_POLY1305_SHA256 (only some OSs, as specified in OSs That Support kTLS)

To verify which TLS ciphers supported by OpenSSL are enabled in your NGINX binary, run the openssl-3.0.0/.openssl/bin/openssl ciphers command in the directory where you built NGINX (for example, your home directory).

Enabling kTLS in NGINX

As mentioned in the introduction, kTLS improves NGINX performance because all encryption and decryption take place in the kernel. Data is encrypted directly in kernel space – before being passed to the network stack for transmission – eliminating the need to copy data into user space to be encrypted by TLS libraries and then back into kernel space for transmission.

Loading kTLS in the Kernel

In modern FreeBSD and Linux distributions, kTLS is usually built as a module (with the CONFIG_TLS=m option). You must explicitly load the kTLS module into the kernel before you start NGINX.

• On FreeBSD, run these commands as the root user:

  # kldload ktls_ocf.ko# sysctl kern.ipc.tls.enable=1
  

  For details about the FreeBSD command options, see the man page for ktls(4).

• On Linux distributions, run this command as the root user:

  # modprobe tls
  

Enabling NGINX with kTLS on FreeBSD

To enable kTLS support in NGINX on FreeBSD, you may use the same instructions as for Linux distributions. However, we recommend that you perform the following steps to leverage the build of NGINX with kTLS in the security/openssl-devel port in the FreeBSD Ports Collection. For more information, including an overview of kTLS, see TLS Offload in the Kernel on the FreeBSD website.

1. Build OpenSSL 3.0 with kTLS support, selecting the appropriate options in the config menu:

   # cd /usr/ports/security/openssl-devel && make config && make install
   

2. Modify /etc/make.conf to use openssl-devel as the default SSL library:

   # echo "DEFAULT_VERSIONS+=ssl=openssl-devel" >> /etc/make.conf
   

3. Build NGINX:

   # cd /usr/ports/www/nginx-devel && make install
   

Building NGINX with kTLS on Linux Distributions

Most current Linux distributions include an OpenSSL version earlier than 3.0.0 (commonly, version 1.1). So you need to build NGINX from source with OpenSSL 3.0.0.

The two crucial options on the configure command that enable kTLS support are:

• --with-openssl=../openssl-3.0.0
• --with-openssl-opt=enable-ktls

The other configure options are for the modules included in the official NGINX binary packages available at nginx.org. You may specify a custom set of modules instead. To see the build options used for your current NGINX binary, run nginx -V.

To build NGINX with OpenSSL 3.0.0, run the following commands:

$ wget https://nginx.org/download/nginx-1.21.4.tar.gz$ wget https://www.openssl.org/source/openssl-3.0.0.tar.gz
$ tar xzf openssl-3.0.0.tar.gz
$ cd nginx-1.21.4
$ ./configure \
--with-debug \
--prefix=/usr/local \
--conf-path=/usr/local/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--http-client-body-temp-path=/var/cache/nginx/client_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
--http-scgi-temp-path=/var/cache/nginx/scgi_temp \
--user=nginx \
--group=nginx \
--with-compat \
--with-file-aio \
--with-threads \
--with-http_addition_module \
--with-http_auth_request_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_mp4_module \
--with-http_random_index_module \
--with-http_realip_module \
--with-http_secure_link_module \
--with-http_slice_module \
--with-http_ssl_module \
--with-http_stub_status_module \
--with-http_sub_module \
--with-http_v2_module \
--with-mail \
--with-mail_ssl_module \
--with-stream \
--with-stream_realip_module \
--with-stream_ssl_module \
--with-stream_ssl_preread_module \
--with-openssl=../openssl-3.0.0 \
--with-openssl-opt=enable-ktls \
--with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' \
-with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
$ make –j4
$ make install

Note: The resulting NGINX binary is statically linked with OpenSSL 3.0.0 libraries. If you later need to patch OpenSSL, you must download and unpack the new OpenSSL source archive, then run the commands above to rebuild the NGINX binary.

Configuring NGINX

To enable kTLS, include the ssl_conf_command directive with the Options KTLS parameter in the server{} context, as in this sample configuration used for our testing:

worker_processes auto;error_log /var/log/nginx/error.log debug;

events {}

http {
    sendfile on;

    server {
        listen 443 ssl;
        ssl_certificate ssl/example.crt;
        ssl_certificate_key ssl/example.key;
        ssl_conf_command Options KTLS;
        ssl_protocols TLSv1.3;

        location / {
            root /data;
    	}
    }
}

Verifying kTLS is Enabled

To verify that NGINX is using kTLS, enable debugging mode and check for BIO_get_ktls_send() and SSL_sendfile() in the error log.

$ grep BIO /var/log/nginx/error.log2021/11/10 16:02:46 [debug] 274550#274550: *2 BIO_get_ktls_send(): 1
2021/11/10 16:02:49 [debug] 274550#274550: *3 BIO_get_ktls_send(): 1
 
$ grep SSL_sendfile /var/log/nginx/error.log
2021/11/10 16:02:46 [debug] 274550#274550: *2 SSL_sendfile: 1048576
2021/11/10 16:02:49 [debug] 274550#274550: *3 SSL_sendfile: 1048576

Note: We recommend that you disable debugging mode after making these checks, especially in production environments. Debug logging incurs a performance penalty due to the large volume of write operations; also, debug logs can be huge and quickly exhaust available space on the disk partition.

Performance Improvement with kTLS

When serving static files and cached responses under heavy load, SSL_sendfile() can increase throughput by up to 2x compared to user‑space TLS, but the size of the performance boost depends significantly on various factors (disk performance, system load, etc). It is also possible to reduce CPU usage if your network card supports TLS offload.

Testing Performance

To measure the performance boost on your setup, use the following instructions to run a simple one‑thread test. As detailed below, our test results indicate a performance boost of up to nearly 30% without any specific tuning.

Hardware and software used:

• AWS t3.medium instance with:
  • 4 GB RAM
  • 20 GB general purpose SSD
  • Intel® Xeon® Platinum 8259CL CPU @ 2.50GHz with 2 cores
• FreeBSD 13.0 and Ubuntu 21.10
• TLSv1.3 with the TLS_AES_256_GCM_SHA384 cipher suite
• NGINX 1.21.4, built and configured as specified in Enabling kTLS in NGINX.

To perform the test:

1. Create a large file that fits completely in the disk cache:

   # truncate -s 1g /data/1G
   

2. Run this command to check the throughput; the base command is repeated multiple times for more accurate results. Pipe the output to the ministat utility[FreeBSD][Ubuntu] for a basic statistical analysis.

    

   # for i in 'seq 1 100'; do curl -k -s -o /dev/null -w '%{speed_download}\n' https://localhost/1G | ministat
   

Results of Performance Testing

In the following results from our tests, presented as output from ministat, each value is the download speed in kBytes/second. The output is split across two lines for legibility.

Throughput for FreeBSD 13.0 without kTLS:

    N           Min           Max        Median     ...x  10        532225        573348        555616     ...  

   ...      Avg        Stddev  
   ... 555155.6     10239.137

Throughput for FreeBSD 13.0 with kTLS:

    N           Min           Max        Median     ...x  10        629379        723164        717349     ...      

   ...      Avg        Stddev
   ... 708600.4     28304.766

Throughput for Ubuntu 21.10 without kTLS:

    N           Min           Max        Median     ...x  10        529199        705720        662354     ...      

   ...      Avg        Stddev
   ... 654321.6     48025.103

Throughput for Ubuntu 21.10 with kTLS:

    N           Min           Max        Median     ...x  10        619105        760208        756278     ...      

   ...      Avg        Stddev
   ... 741848.3     43255.246

In our testing, kTLS boosted performance more with FreeBSD than Ubuntu. The percentage improvement was as follows:

Summary

NGINX 1.21.4 introduces support for kTLS when serving static files and cached responses with SSL_sendfile(). Our testing shows that performance improves by between 8% and 29%, depending on the operating system.

We’re interested in hearing about your experiences with kTLS and NGINX, and especially the results of your testing on other OSs! Please share them in the comments section below.