Recovering from a Failed NGINX Plus Upgrade: 'module "M" version X instead of Y'

If you are using dynamic modules, you might see an error like this during the upgrade to a new NGINX or NGINX Plus release:

Setting up nginx-plus (1.11.10-1~xenial) ...
nginx: [emerg] module "/etc/nginx/modules/ngx_http_geoip_module.so" version 1011005 instead of 1011006 in /etc/nginx/nginx.conf:7

nginx: configuration file /etc/nginx/nginx.conf test failed

invoke-rc.d: initscript nginx, action "upgrade" failed.

The most likely reason is that you haven’t upgraded the specified dynamic module (.so file):

• If you are running NGINX Open Source, dynamic modules must be compiled against the version you are upgrading to.
• If you are running NGINX Plus, dynamic modules must be compiled against the NGINX Open Source version that corresponds to the NGINX Plus release you are upgrading to.

For the sake of brevity we’ll refer to NGINX Plus only from now on.

Don’t Panic!

Rest assured that this error message does not indicate that your NGINX Plus server is down. NGINX Plus upgrades are seamless, so the old version continues running when an upgrade fails. However, the upgrade is incomplete and your system is in an inconsistent state:

• NGINX Plus processes in memory are running the old version
• All of the files on disk relating to NGINX Plus correspond to the new version
• The dynamic module mentioned in the error message is compiled against the old version

Do not manually restart nginx, or reboot the system. Doing so will cause downtime because new NGINX Plus processes cannot start while there are dynamic modules out of sync with the new NGINX Plus release.

Note that the error message relates only to the first incompatible module found during the upgrade process, so it is important to find all load_module directives in your configuration and ensure that every module is compiled against the appropriate NGINX version. Correct versions of all NGINX‑authored and certified third‑party dynamic modules are provided for each NGINX Plus release<, so you need to recompile only non‑certified modules.This article describes how to safely complete the upgrade process.

What Went Wrong?

NGINX 1.11.5 and NGINX Plus R11 introduced the ability to compile dynamic modules against NGINX Open Source and load them into NGINX Plus. This binary compatibility requires the module and NGINX Plus to share the same base open source version. Upgrading NGINX Plus without first installing dynamic modules built against the corresponding NGINX Open Source version fails because of this version mismatch.

If your dynamic module was supplied by a third‑party vendor then you need to contact the vendor for a new version, matched to the NGINX version for the NGINX Plus release you intend to upgrade to. If your dynamic module was compiled from source (and you have access to the sources) then continue reading.

The Road to Recovery

Step 0: Prepare the Build Environment

We strongly recommend that you compile dynamic modules on a separate system, which we refer to here as the “build environment”. Doing so minimizes the risk and complexity of the system where you are running NGINX Plus with the dynamic module (we refer to this as the “production environment”). The build environment must have the same operating system and version as the production environment; in addition, the following components must be installed:

• UnZip utility
• Compiler and make utility
• Perl‑compatible regular expressions library (development files)
• Zlib compression libraries (development files)

To ensure your build environment has these prerequisites installed, run the following command.

• For Ubuntu/Debian:

  buildenv$ sudo apt-get install unzip gcc make libpcre3-dev zlib1g-dev

• For CentOS/RHEL/Oracle Linux:

  buildenv$ sudo yum install unzip gcc make pcre-devel zlib-devel

Step 1: Obtain NGINX Open Source

1. Working in the production environment, run the following command to identify the NGINX Open Source version that corresponds to the running NGINX Plus release. It’s highlighted in orange in this output: NGINX 1.11.10, which corresponds to NGINX Plus R12.

   production$ nginx -vnginx version: nginx/1.11.10 (nginx-plus-r12)

2. Working in the build environment, download the sources for the appropriate NGINX Open Source version.

   build-env$ wget -qO - http://nginx.org/download/nginx-1.11.10.tar.gz | tar zxfv -

Step 2: Obtain the Dynamic Module Sources

Copy the source code for the dynamic module to the build directory of your choice. Here we copy a sample NGINX “hello world” module from its GitHub repository.

buildenv$ git clone https://github.com/perusio/nginx-hello-world-module.git
Cloning into 'nginx-hello-world-module'...

Step 3: Compile the Dynamic Module

1. Compile the dynamic module by first running the NGINX configure script with the --with-compat argument to make the dynamic module binary‑compatible with NGINX Plus. Then run make modules to compile just the module.

   buildenv$ cd nginx-1.11.10/buildenv$ ./configure --with-compat --add-dynamic-module=../nginx-hello-world-module
   buildenv$ make modules

2. Verify that the build process has created the dynamic module as a .so file in the objs subdirectory.

   buildenv$ ls objs/*.soobjs/ngx_http_hello_world.so

3. Create a copy of the module file with the NGINX Open Source version in the filename. This makes it simpler to manage multiple versions of the dynamic module in the production environment.

   buildenv$ cp objs/ngx_http_hello_world.so ./ngx_http_hello_world_1.11.10.so

Step 4: Copy the Dynamic Module to the Production Environment

Because we created the dynamic module with the NGINX Open Source version in the filename, we can safely copy it to the production environment without affecting operation.

buildenv$ scp ./ngx_http_hello_world_1.11.10.so production:/etc/nginx/modules

Step 5: Swap Over the Dynamic Module

Working in the production environment, swap in the new .so file to replace the current one. This is safe to do at this point because dynamic modules are loaded into memory only when nginx restarts or the configuration is reloaded.

production$ cd /etc/nginx/modulesproduction$ cp ngx_http_hello_world.so ngx_http_hello_world_ROLLBACK.so
production$ ln -fs ngx_http_hello_world_1.11.10.so ngx_http_hello_world.so
production$ ls -gG
-rw-r--r-- 1 243576 Jan 31 16:18 ngx_http_hello_world_1.11.10.so
-rw-r--r-- 1 243576 Oct 20 10:40 ngx_http_hello_world_ROLLBACK.so
lrwxrwxrwx 1     24 Jan 31 16:26 ngx_http_hello_world.so -> ngx_http_hello_world_1.11.10.so

Step 6: Complete the Upgrade of NGINX Plus

1. Run the following command to test that the module version is correct and that NGINX Plus is ready to complete the upgrade process.

   production$ nginx -tnginx: the configuration file /etc/nginx/nginx.conf syntax is ok
   nginx: configuration file /etc/nginx/nginx.conf test is successful

2. Complete the upgrade process so that NGINX Plus is using the new release and dynamic modules.

   production$ service nginx upgradeStarting new master nginx:                                 [  OK  ]
   Graceful shutdown of old nginx:                            [  OK  ]

Preventing the Upgrade Failure

The next time you upgrade NGINX Plus, you can save time and avoid errors by following these instructions before the upgrade so that updated dynamic modules are already available when you start the upgrade process.