Announcing NGINX Plus R28

We’re happy to announce the availability of NGINX Plus Release 28 (R28). Based on NGINX Open Source, NGINX Plus is the only all-in-one software web server, load balancer, reverse proxy, content cache, and API gateway.

New and enhanced features in NGINX Plus R28 include:

• Additional TLS metrics – NGINX Plus R28 collects additional TLS statistics at the system‑wide, client‑side, and server‑side levels, providing insight that’s critical when troubleshooting SSL/TLS‑related errors in proxy configuration and for connections to clients and upstream servers.

  The NGINX Plus live activity monitoring dashboard is updated to display the new SSL session data.

• Support for PROXY protocol v2 TLVs in cloud private services – When you make resources available to external clients via a “private service” in AWS, Google Cloud Platform, and Microsoft Azure, by default the service‑specific client identifier represented in a type-length-value (TLV) vector in the PROXY protocol v2 header is not passed to backend services. NGINX Plus R28 introduces modules for the http and stream contexts which decode the TLV and define a variable for forwarding the client identifier to backend services.
• Variable support for the sticky cookie samesite parameter – In NGINX Plus R28, the value of the samesite parameter to the sticky cookie directive can be a variable. This improvement allows for easier management of traffic and more security.

Rounding out the release are new features and bug fixes inherited from NGINX Open Source and updates to the NGINX JavaScript module.

Important Changes in Behavior

Note: If you are upgrading from a release other than NGINX Plus R27, be sure to check the Important Changes in Behavior section in the announcement blogs for all releases between your current one and this one.

Changes to Platform Support

New operating systems and architectures supported:

• AlmaLinux 8 and 9 (x86_64, aarch64)
• Alpine 3.17 (x86_64, arm64)
• Oracle Linux 9 (x86_64)
• Rocky Linux 8 and 9 (x86_64, aarch64)

Older operating systems removed:

• Debian 10, which reached end of life (EOL) in August 2022

Older operating systems and architectures deprecated and scheduled for removal in NGINX Plus R29:

• Alpine 3.13, which reached EOL on November 1, 2022

Change in handling multiple headers with identical names

• Most of known duplicate upstream response headers are now ignored with a warning
• Duplicate Content-Length and Transfer-Encoding headers are now rejected as well as the responses with invalid Content-Length or Transfer-Encoding headers, or if both Content-Length and Transfer-Encoding are present in the response

New Features in Detail

Additional TLS Metrics

Observability of SSL/TLS events and errors is important when troubleshooting TLS‑related problems with proxy configuration, upstream servers, and clients. Since the introduction of the NGINX Plus API in NGINX Plus R13, NGINX Plus has collected three TLS metrics at the system‑wide level:

• handshakes – Number of successful SSL handshakes
• handshakes_failed – Number of SSL handshake failures (which do not include certificate verification failures that happen after the SSL handshake)
• session_reuses – Number of SSL session reuses

In NGINX Plus R27<.htmla> and later, you can configure collection of the three metrics for individual upstream servers and virtual servers.

NGINX Plus R28 expands the set of TLS metrics with new counters for handshake errors and certificate validation failures in both HTTP and Stream modules (here we provide examples only for HTTP modules, but the available Stream metrics are similar). For details on configuring metrics collection for individual upstream servers and virtual servers, see the NGINX Plus R27<.htmlspan> announcement blog.

Handshake Errors

Counters for the following handshake errors are new in NGINX Plus R28:

• handshake_timeout – Number of handshake failures due to handshake timeout
• no_common_cipher – Number of handshake failures due to lack of a common cipher between the parties in the handshake (not collected for connections to upstream servers because it does not apply)
• no_common_protocol – Number of handshake failures due to lack of a common protocol between the parties
• peer_rejected_cert – Number of handshake failures due to the other party rejecting the certificate presented by NGINX Plus and providing the proper alert message

Certificate Verification Failures

Certificate verification failures are now reported in the new verify_failures section of API output when you configure certificate verification:

• For connections to clients with the ssl_verify_client[HTTP][Stream] directive

• For connections to servers with these protocol‑specific directives:

  • grpc_ssl_verify
  • proxy_ssl_verify[HTTP][Stream]
  • uwsgi_ssl_verify

When a certificate verification failure occurs, the metric for the corresponding cause is incremented and the connection torn down. Note, however, that the basic handshakes counter is still incremented because these failures occur after the handshake succeeds.

The metrics for failed certificate validation are:

• expired_cert – Peer presented an expired certificate
• hostname_mismatch – Server’s certificate doesn’t match the hostname (not collected for connections to clients)
• no_cert – Client didn’t provide a certificate as required (not collected for connections to upstream servers)
• revoked_cert – Peer presented a revoked certificate
• other – Explicit counter for other certificate verification failures

Sample Metrics Output

Here’s a set of sample TLS metrics for HTTP connections at the system‑wide level:

$ curl 127.0.0.1:8080/api/8/ssl{
    "handshakes": 32,
    "session_reuses": 0,
    "handshakes_failed": 8,
    "no_common_protocol": 4,
    "no_common_cipher": 2,
    "handshake_timeout": 0,
    "peer_rejected_cert": 0,
    "verify_failures": {
        "no_cert": 0,
        "expired_cert": 2,
        "revoked_cert": 1,
        "hostname_mismatch": 2,
        "other": 1
    }
}

Here’s a set of sample metrics for connections between clients and the HTTP virtual server s9 (as noted previously, the hostname_mismatch counter is not collected for such connections):

$ curl 127.0.0.1:8080/api/8/http/server_zones/s9{
    ...
    "ssl": {
        "handshakes": 0,
        "session_reuses": 0,
        "handshakes_failed": 1,
        "no_common_protocol": 0,
        "no_common_cipher": 1,
        "handshake_timeout": 0,
        "peer_rejected_cert": 0,
        "verify_failures": {
            "no_cert": 0,
            "expired_cert": 0,
            "revoked_cert": 0,
            "other": 0
        }
    }
    ...
}

Here’s a set of sample metrics for HTTP connections to servers in the u2 upstream group (as noted previously, the no_cert and no_common_cipher counters are not collected for such connections):

$ curl 127.0.0.1:8080/api/8/http/upstreams/u2{
    "peers": [
        {
            "id": 0,
            "server": "127.0.0.1:8082",
            "name": "127.0.0.1:8082",
            ...
            "ssl": {
                "handshakes": 1,
                "session_reuses": 0,
                "handshakes_failed": 0,
                "no_common_protocol": 0,
                "handshake_timeout": 0,
                "peer_rejected_cert": 0,
                "verify_failures": {
                    "expired_cert": 1,
                    "revoked_cert": 0,
                    "hostname_mismatch": 0,
                    "other": 0
                }
            },
            ...
        }
    ],
}

NGINX Plus Dashboard Displays the Extended TLS Metrics

For NGINX Plus R28 and later, the live activity monitoring dashboard displays the new TLS metrics described above. This screenshot shows metrics for connections to clients. To view the new metrics, mouse over the value in the SSL > Handshakes failed column as shown.

Support for PROXY Protocol v2 TLVs in Cloud Private Services

The three leading cloud providers – Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure – each offer a “private service” where you can enable external clients to access your services without exposing them on the public Internet. Each service makes use of a client identifier which is represented in a type-length-value (TLV) vector in a PROXY protocol v2 header. The service‑specific identifiers are:

• AWS PrivateLink – PP2_SUBTYPE_AWS_VPCE_ID
• GCP Private Service Connect – pscConnectionId
• Microsoft Azure Private Link – PP2_SUBTYPE_AZURE_PRIVATEENDPOINT_LINKID

By default these client identifiers are not passed to backend services. NGINX Plus R28 introduces modules for the http and stream contexts – ngx_http_proxy_protocol_vendor_module and ngx_stream_proxy_protocol_vendor_module – which decode the TLV and define a variable for forwarding the identifier to backend services.

For general information about how NGINX Plus uses the PROXY protocol to obtain IP addresses and other information about clients, see Accepting the PROXY Protocol in the NGINX Plus Admin Guide.

PROXY Protocol v2 Support for AWS

In AWS, the source IP address for traffic coming from clients through a Virtual Private Cloud (VPC) endpoint service is the private IP address of the Network Load Balancer node. If the backend application requires the real IP addresses and other identifiers for clients, they can be obtained from PROXY protocol v2 headers.

In AWS, a custom TLV vector encodes the VPC ID of the endpoint in the PROXY protocol v2 header PP2_SUBTYPE_AWS_VPCE_ID. (For more information, see the AWS documentation.)

NGINX Plus R28 decodes the TLV and passes the endpoint ID to backend applications in the $proxy_protocol_tlv_aws_vpce_id variable.

Note: In the server block where you reference the $proxy_protocol_tlv_aws_vpce_id variable, you must also include the proxy_protocol parameter to the listen[HTTP][Stream] directive. For an example, see line 8 of proxy_protocol_v2.conf just below.

This sample configuration for AWS checks that the VPC ID is acceptable and if so passes it to the backend application as the second parameter to the add_header directive:

PROXY Protocol v2 Support for GCP

In GCP Private Service Connect, the source IP address for traffic coming from clients is an “address in one of the Private Service Connect subnets in the service producer’s VPC network”. If the backend application requires the real IP addresses and other identifiers for clients, they can be obtained from PROXY protocol v2 headers.

In GCP, a custom TLV vector encodes the unique (at the time) connection ID in the PROXY protocol v2 header pscConnectionId. (For more information, see the GCP documentation.)

NGINX Plus R28 decodes the TLV and passes the value of pscConnectionId to backend applications in the $proxy_protocol_tlv_gcp_conn_id variable.

Note: In the server block where you reference the $proxy_protocol_tlv_gcp_conn_id variable, you must also include the proxy_protocol parameter to the listen[HTTP][Stream] directive. For an example, see line 8 of proxy_protocol_v2.conf above.

PROXY Protocol v2 Support for Microsoft Azure

In Microsoft Azure Private Link, the source IP address for traffic coming from clients is the “network address translated (NAT) on the service provider side using the NAT IP [address] allocated from the provider’s virtual network”. If the backend application requires the real IP addresses and other identifiers for clients, they can be obtained from PROXY protocol v2 headers.

In Azure, a custom TLV vector encodes the client’s LinkID in the PROXY protocol v2 header PP2_SUBTYPE_AZURE_PRIVATEENDPOINT_LINKID. (For more information, see the Azure documentation.)

NGINX Plus R28 decodes the TLV and passes the LinkID to backend applications in the $proxy_protocol_tlv_azure_pel_id variable.

Note: In the server block where you reference the $proxy_protocol_tlv_azure_pel_id variable, you must also include the proxy_protocol parameter to the listen[HTTP][Stream] directive. For an example, see line 8 of proxy_protocol_v2.conf above.

Variable Support for the Sticky Cookie samesite Parameter

In previous NGINX Plus releases, three static values (strict, lax, and none) were acceptable for the samesite parameter to the sticky cookie directive. In NGINX Plus R28, the value can also be a variable.

By default (there is no samesite parameter), NGINX does not inject the SameSite attribute into the cookie. When the samesite parameter is a variable, the result depends on how the variable resolves at runtime:

• To one of standard values (strict, lax, and none) – NGINX injects the SameSite attribute set to that value
• To an empty value ("") – NGINX does not inject the SameSite attribute
• To any other value, which indicates a misconfiguration – NGINX injects the SameSite attribute set to Strict (the most secure setting)

This sample configuration sets the samesite attribute based on the value of the HTTP User-Agent header (this is good for legacy clients that don’t support the SameSite attribute):

Other Enhancements in NGINX Plus R28

Changes Inherited from NGINX Open Source

NGINX Plus R28 is based on NGINX Open Source 1.23.2. and inherits functional changes and bug fixes made since NGINX Plus R27 was released (in NGINX 1.23.0 through 1.23.2). Changes and bug fixes include:

• The new ipv4=off parameter to the HTTP resolver directive disables lookup of IPv4 addresses.
• When you enable a shared cache for HTTP session information with the shared parameter to the ssl_session_cache directive, TLS session ticket keys are now rotated automatically.
• The severity level at which several types of TLS/SSL errors are logged is lowered from crit to info.

For the full list of new features, changes, and bug fixes inherited from these releases, see the CHANGES file.

Changes to the NGINX JavaScript Module

NGINX Plus R28 incorporates changes and fixes made in versions 0.7.5 through 0.7.8 of the NGINX JavaScript module (njs). We have highlighted some of the most significant ones in Make Your NGINX Config Even More Modular and Reusable with njs 0.7.7 on our blog. For a full list, see the Changes file.

Upgrade or Try NGINX Plus

If you’re running NGINX Plus, we strongly encourage you to upgrade to NGINX Plus R28 as soon as possible. You’ll also pick up several additional fixes and improvements, and it will help NGINX to help you when you need to raise a support ticket.

If you haven’t tried NGINX Plus, we encourage you to try it out – for security, load balancing, and API gateway, or as a fully supported web server with enhanced monitoring and management APIs. You can get started today with a free 30-day trial.