BLOG | NGINX

NGINX Tutorial: How to Securely Manage Secrets in Containers

NGINX-Part-of-F5-horiz-black-type-RGB
Robert Haynes Thumbnail
Robert Haynes
Published March 14, 2023

This post is one of four tutorials that help you put into practice concepts from Microservices March 2023: Start Delivering Microservices:

Many of your microservices need secrets to operate securely. Examples of secrets include the private key for an SSL/TLS certificate, an API key to authenticate to another service, or an SSH key for remote login. Proper secrets management requires strictly limiting the contexts where secrets are used to only the places they need to be and preventing secrets from being accessed except when needed. But this practice is often skipped in the rush of application development. The result? Improper secrets management is a common cause of information leakage and exploits.

Tutorial Overview

In this tutorial, we show how to safely distribute and use a JSON Web Token (JWT) which a client container uses to access a service. In the four challenges in this tutorial, you experiment with four different methods for managing secrets, to learn not only how to manage secrets correctly in your containers but also about methods that are inadequate:

Although this tutorial uses a JWT as a sample secret, the techniques apply to anything for containers that you need to keep secret, such as database credentials, SSL private keys, and other API keys.

The tutorial leverages two main software components:

  • API server – A container running NGINX Open Source and some basic NGINX JavaScript code that extracts a claim from the JWT and returns a value from one of the claims or, if no claim is present, an error message
  • API client – A container running very simple Python code that simply makes a GET request to the API server

Watch this video for a demo of the tutorial in action.

The easiest way to do this tutorial is to register for Microservices March and use the browser‑based lab that’s provided. This post provides instructions for running the tutorial in your own environment.

Prerequisites and Set Up

Prerequisites

To complete the tutorial in your own environment, you need:

Notes:

  • The tutorial makes use of a test server listening on port 80. If you’re already using port 80, use the ‑p flag to set a different value for the test server when you start it with the docker run command. Then include the :<port_number> suffix on localhost in the curl commands.
  • Throughout the tutorial the prompt on the Linux command line is omitted, to make it easier to cut and paste the commands into your terminal. The tilde (~) represents your home directory.

Set Up

In this section you clone the tutorial repo, start the authentication server, and send test requests with and without a token.

Clone the Tutorial Repo

  1. In your home directory, create the microservices-march directory and clone the GitHub repository into it. (You can also use a different directory name and adapt the instructions accordingly.) The repo includes config files and separate versions of the API client application that use different methods to obtain secrets.

    mkdir ~/microservices-marchcd ~/microservices-march
git clone https://github.com/microservices-march/auth.git

  2. Display the secret. It’s a signed JWT, commonly used to authenticate API clients to servers.

    cat ~/microservices-march/auth/apiclient/token1.jwt"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6InNpZ24ifQ.eyJpYXQiOjE2Nz UyMDA4MTMsImlzcyI6ImFwaUtleTEiLCJhdWQiOiJhcGlTZXJ2aWNlIiwic3ViIjoiYXBpS2V5MSJ9._6L_Ff29p9AWHLLZ-jEZdihy-H1glooSq_z162VKghA"

While there are a few ways to use this token for authentication, in this tutorial the API client app passes it to the authentication server using the OAuth 2.0 Bearer Token Authorization framework. That involves prefixing the JWT with Authorization: Bearer as in this example:

"Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6InNpZ24ifQ.eyJpYXQiOjE2NzUyMDA4MTMsImlzcyI6ImFwaUtleTEiLCJhdWQiOiJhcGlTZXJ2aWNlIiwic3ViIjoiYXBpS2V5MSJ9._6L_Ff29p9AWHLLZ-jEZdihy-H1glooSq_z162VKghA"

Build and Start the Authentication Server

  1. Change to the authentication server directory:

    cd apiserver

  2. Build the Docker image for the authentication server (note the final period):

    docker build -t apiserver .

  3. Start the authentication server and confirm that it’s running (the output is spread over multiple lines for legibility):

    docker run -d -p 80:80 apiserver
docker ps
CONTAINER ID   IMAGE       COMMAND                  ...
2b001f77c5cb   apiserver   "nginx -g 'daemon of..." ...  


    ... CREATED         STATUS          ...                                    
    ... 26 seconds ago  Up 26 seconds   ... 


    ... PORTS                                      ...
    ... 0.0.0.0:80->80/tcp, :::80->80/tcp, 443/tcp ...


    ... NAMES
    ... relaxed_proskuriakova

Test the Authentication Server

  1. Verify that the authentication server rejects a request that doesn’t include the JWT, returning 401 Authorization Required:

    curl -X GET http://localhost<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.23.3</center>
</body>
</html>

  2. Provide the JWT using the Authorization header. The 200 OK return code indicates the API client app authenticated successfully.

    curl -i -X GET -H "Authorization: Bearer `cat $HOME/microservices-march/auth/apiclient/token1.jwt`" http://localhostHTTP/1.1 200 OK
Server: nginx/1.23.2
Date: Day, DD Mon YYYY hh:mm:ss TZ
Content-Type: text/html
Content-Length: 64
Last-Modified: Day, DD Mon YYYY hh:mm:ss TZ
Connection: keep-alive
ETag: "63dc0fcd-40"
X-MESSAGE: Success apiKey1
Accept-Ranges: bytes


{ "response": "success", "authorized": true, "value": "999" }

Challenge 1: Hardcode Secrets in Your App (Not!)

Before you begin this challenge, let’s be clear: hardcoding secrets into your app is a terrible idea! You’ll see how anyone with access to the container image can easily find and extract hardcoded credentials.

In this challenge, you copy the code for the API client app into the build directory, build and run the app, and extract the secret.

Copy the API Client App

The app_versions subdirectory of the apiclient directory contains different versions of the simple API client app for the four challenges, each slightly more secure than the previous one (see Tutorial Overview for more information).

  1. Change to the API client directory:

    cd ~/microservices-march/auth/apiclient

  2. Copy the app for this challenge – the one with a hardcoded secret – to the working directory:

    cp ./app_versions/very_bad_hard_code.py ./app.py

  3. Take a look at the app:

    cat app.pyimport urllib.request
import urllib.error

jwt = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6InNpZ24ifQ.eyJpYXQiOjE2NzUyMDA4MTMsImlzcyI6ImFwaUtleTEiLCJhdWQiOiJhcGlTZXJ2aWNlIiwic3ViIjoiYXBpS2V5MSJ9._6L_Ff29p9AWHLLZ-jEZdihy-H1glooSq_z162VKghA"
authstring = "Bearer " + jwt
req = urllib.request.Request("http://host.docker.internal")
req.add_header("Authorization", authstring)
try:
    with urllib.request.urlopen(req) as response:
        the_page = response.read()
        message = response.getheader("X-MESSAGE")
        print("200  " + message)
except urllib.error.URLError as e:
    print(str(e.code) + " s " + e.msg)

    The code simply makes a request to a local host and prints out either a success message or failure code.

    The request adds the Authorization header on this line:

    req.add_header("Authorization", authstring)

    Do you notice anything else? Perhaps a hardcoded JWT? We will get to that in a minute. First let’s build and run the app.

Build and Run the API Client App

We’re using the docker compose command along with a Docker Compose YAML file – this makes it a little easier to understand what’s going on.

(Notice that in Step 2 of the previous section you renamed the Python file for the API client app that’s specific to Challenge 1 (very_bad_hard_code.py) to app.py. You’ll also do this in the other three challenges. Using app.py each time simplifies logistics because you don’t need to change the Dockerfile. It does mean that you need to include the ‑build argument on the docker compose command to force a rebuild of the container each time.)

The docker compose command builds the container, starts the application, makes a single API request, and then shuts down the container, while displaying the results of the API call on the console.

The 200 Success code on the second-to-last line of the output indicates that authentication succeeded. The apiKey1 value is further confirmation, because it shows the auth server was able to decode the claim of that name in the JWT:

docker compose -f docker-compose.hardcode.yml up -build
...
apiclient-apiclient-1  | 200  Success apiKey1
apiclient-apiclient-1 exited with code 0

So hardcoded credentials worked correctly for our API client app – not surprising. But is it secure? Maybe so, since the container runs this script just once before it exits and doesn’t have a shell?

In fact – no, not secure at all.

Retrieve the Secret from the Container Image

Hardcoding credentials leaves them open to inspection by anyone who can access the container image, because extracting the filesystem of a container is a trivial exercise.

  1. Create the extract directory and change to it:

    mkdir extractcd extract

  2. List basic information about the container images. The --format flag makes the output more readable (and the output is spread across two lines here for the same reason):

    docker ps -a --format "table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.RunningFor}}\t{{.Status}}"CONTAINER ID   NAMES                   IMAGE       ...
11b73106fdf8   apiclient-apiclient-1   apiclient   ...
ad9bdc05b07c   exciting_clarke         apiserver   ...


    ... CREATED          STATUS
    ... 6 minutes ago    Exited (0) 4 minutes ago
    ... 43 minutes ago   Up 43 minutes

  3. Extract the most recent apiclient image as a .tar file. For <container_ID>, substitute the value from the CONTAINER ID field in the output above (11b73106fdf8 in this tutorial):

    docker export -o api.tar <container_ID>

    It takes a few seconds to create the api.tar archive, which includes the container’s entire file system. One approach to finding secrets is to extract the whole archive and parse it, but as it turns out there is a shortcut for finding what’s likely to be interesting – displaying the container’s history with the docker history command. (This shortcut is especially handy because it also works for containers that you find on Docker Hub or another container registry and thus might not have the Dockerfile, but only the container image).

  4. Display the history of the container:

    docker history apiclient
IMAGE         CREATED        ...
9396dde2aad0  8 minutes ago  ...                    
<missing>     8 minutes ago  ...   
<missing>     28 minutes ago ...  
               
    ... CREATED BY                          SIZE ... 
    ... CMD ["python" "./app.py"]           622B ...   
    ... COPY ./app.py ./app.py # buildkit   0B   ... 
    ... WORKDIR /usr/app/src                0B   ...   
             
    ... COMMENT
    ... buildkit.dockerfile.v0
    ... buildkit.dockerfile.v0
    ... buildkit.dockerfile.v0

    The lines of output are in reverse chronological order. They show that the working directory was set to /usr/app/src, then the file of Python code for the app was copied in and run. It doesn’t take a great detective to deduce that the core codebase of this container is in /usr/app/src/app.py, and as such that’s a likely location for credentials.

  5. Armed with that knowledge, extract just that file:

    tar --extract --file=api.tar usr/app/src/app.py

  6. Display the file’s contents and, just like that, we have gained access to the “secure” JWT:

    cat usr/app/src/app.py
...
jwt="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6InNpZ24ifQ.eyJpYXQiOjE2NzUyMDA4MTMsImlzcyI6ImFwaUtleTEiLCJhdWQiOiJhcGlTZXJ2aWNlIiwic3ViIjoiYXBpS2V5MSJ9._6L_Ff29p9AWHLLZ-jEZdihy-H1glooSq_z162VKghA"
...

Challenge 2: Pass Secrets as Environment Variables (Again, No!)

If you completed Unit 1 of Microservices March 2023 (Apply the Twelve‑Factor App to Microservices Architectures), you’re familiar with using environment variables to pass configuration data to containers. If you missed it, never fear – it’s available on demand after you register.

In this challenge, you pass secrets as environment variables. Like the method from Challenge 1, we don’t recommend this one! It’s not as bad as hardcoding secrets, but as you’ll see it has some weaknesses.

There are four ways to pass environment variables to a container:

  • Use the ENV statement in a Dockerfile to do variable substitution (set the variable for all images built). For example:

    ENV PORT $PORT

  • Use the ‑e flag on the docker run command. For example:

    docker run -e PASSWORD=123 mycontainer
  • Use the environment key in a Docker Compose YAML file.
  • Use a .env file containing the variables.

In this challenge, you use an environment variable to set the JWT and examine the container to see if the JWT is exposed.

Pass an Environment Variable

  1. Change back to the API client directory:

    cd ~/microservices-march/auth/apiclient

  2. Copy the app for this challenge – the one that uses environment variables – to the working directory, overwriting the app.py file from Challenge 1:

    cp ./app_versions/medium_environment_variables.py ./app.py

  3. Take a look at the app. In the relevant lines of output, the secret (JWT) is read as an environment variable in the local container:

    cat app.py...
jwt = ""
if "JWT" in os.environ:
    jwt = "Bearer " + os.environ.get("JWT")
...

  4. As explained above, there’s a choice of ways to get the environment variable into the container. For consistency, we’re sticking with Docker Compose. Display the contents of the Docker Compose YAML file, which uses the environment key to set the JWT environment variable:

    cat docker-compose.env.yml---
version: "3.9"
services:
  apiclient:
    build: .
    image: apiclient
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
      - JWT

  5. Run the app without setting the environment variable. The 401 Unauthorized code on the second-to-last line of the output confirms that authentication failed because the API client app didn’t pass the JWT:

    docker compose -f docker-compose.env.yml up -build...
apiclient-apiclient-1  | 401  Unauthorized
apiclient-apiclient-1 exited with code 0

  6. For simplicity, set the environment variable locally. It’s fine to do that at this point in the tutorial, since it’s not the security issue of concern right now:

    export JWT=`cat token1.jwt`

  7. Run the container again. Now the test succeeds, with the same message as in Challenge 1:

    docker compose -f docker-compose.env.yml up -build
... 
apiclient-apiclient-1  | 200  Success apiKey1
apiclient-apiclient-1 exited with code 0

So at least now the base image doesn’t contain the secret and we can pass it at run time, which is safer. But there is still a problem.

Examine the Container

  1. Display information about the container images to get the container ID for the API client app (the output is spread across two lines for legibility):

    docker ps -a --format "table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.RunningFor}}\t{{.Status}}"CONTAINER ID   NAMES                   IMAGE      ...
6b20c75830df   apiclient-apiclient-1   apiclient  ...
ad9bdc05b07c   exciting_clarke         apiserver  ...


    ... CREATED             STATUS
    ... 6 minutes ago       Exited (0) 6 minutes ago
    ... About an hour ago   Up About an hour

  2. Inspect the container for the API client app. For <container_ID>, substitute the value from the CONTAINER ID field in the output above (here 6b20c75830df).

    The docker inspect command lets you inspect all launched containers, whether they are currently running or not. And that’s the problem – even though the container is not running, the output exposes the JWT in the Env array, insecurely saved in the container config.

    docker inspect <container_ID>...
"Env": [
  "JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6InNpZ24ifQ.eyJpYXQiOjE2NzUyMDA...",
  "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "LANG=C.UTF-8",
  "GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D",
  "PYTHON_VERSION=3.11.2",
  "PYTHON_PIP_VERSION=22.3.1",
  "PYTHON_SETUPTOOLS_VERSION=65.5.1",
  "PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/1a96dc5acd0303c4700e026...",
  "PYTHON_GET_PIP_SHA256=d1d09b0f9e745610657a528689ba3ea44a73bd19c60f4c954271b790c..."
]

Challenge 3: Use Local Secrets

By now you’ve learned that hardcoding secrets and using environment variables is not as safe as you (or your security team) need it to be.

To improve security, you can try using local Docker secrets to store sensitive information. Again, this isn’t the gold‑standard method, but it’s good to understand how it works. Even if you don’t use Docker in production, the important takeaway is how you can make it difficult to extract the secret from a container.

In Docker, secrets are exposed to a container via the file system mount /run/secrets/ where there’s a separate file containing the value of each secret.

In this challenge you pass a locally stored secret to the container using Docker Compose, then verify that the secret isn’t visible in the container when this method is used.

Pass a Locally Stored Secret to the Container

  1. As you might expect by now, you start by changing to the apiclient directory:

    cd ~/microservices-march/auth/apiclient

  2. Copy the app for this challenge – the one that uses secrets from within a container – to the working directory, overwriting the app.py file from Challenge 2:

    cp ./app_versions/better_secrets.py ./app.py

  3. Take a look at the Python code, which reads the JWT value from the /run/secrets/jot file. (And yes, we should probably be checking that the file only has one line. Maybe in Microservices March 2024?)

    cat app.py...
jotfile = "/run/secrets/jot"
jwt = ""
if os.path.isfile(jotfile):
    with open(jotfile) as jwtfile:
        for line in jwtfile:
            jwt = "Bearer " + line
...

    OK, so how are we going to create this secret? The answer is in the docker-compose.secrets.yml file.

  4. Take a look at the Docker Compose file, where the secret file is defined in the secrets section and then referenced by the apiclient service:

    cat docker-compose.secrets.yml---
version: "3.9"
secrets:
  jot:
    file: token1.jwt
services:
  apiclient:
    build: .
    extra_hosts:
      - "host.docker.internal:host-gateway"
    secrets:
      - jot

Verify the Secret Isn’t Visible in the Container

  1. Run the app. Because we’ve made the JWT accessible within the container, authentication succeeds with the now‑familiar message:

    docker compose -f docker-compose.secrets.yml up -build...
apiclient-apiclient-1  | 200 Success apiKey1
apiclient-apiclient-1 exited with code 0

  2. Display information about the container images, noting the container ID for the API client app (for sample output, see Step 1 in Examine the Container from Challenge 2):

    docker ps -a --format "table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.RunningFor}}\t{{.Status}}"

  3. Inspect the container for the API client app. For <container_ID>, substitute the value from the CONTAINER ID field in the output from the previous step. Unlike the output in Step 2 of Examine the Container, there is no JWT= line at the start of the Env section:

    docker inspect <container_ID>
"Env": [
  "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "LANG=C.UTF-8",
  "GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D",
  "PYTHON_VERSION=3.11.2",
  "PYTHON_PIP_VERSION=22.3.1",
  "PYTHON_SETUPTOOLS_VERSION=65.5.1",
  "PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/1a96dc5acd0303c4700e026...",
  "PYTHON_GET_PIP_SHA256=d1d09b0f9e745610657a528689ba3ea44a73bd19c60f4c954271b790c..."
]

    So far, so good, but our secret is in the container filesystem at /run/secrets/jot. Maybe we can extract it from there using the same method as in Retrieve the Secret from the Container Image from Challenge 1.

  4. Change to the extract directory (which you created during Challenge 1) and export the container into a tar archive:

    cd extractdocker export -o api2.tar <container_ID>

  5. Look for the secrets file in the tar file:

    tar tvf api2.tar | grep jot-rwxr-xr-x  0 0      0           0 Mon DD hh:mm run/secrets/jot

    Uh oh, the file with the JWT in it is visible. Didn’t we say embedding secrets in the container was “secure”? Are things just as bad as in Challenge 1?

  6. Let’s see – extract the secrets file from the tar file and look at its contents:

    tar --extract --file=api2.tar run/secrets/jotcat run/secrets/jot

    Good news! There’s no output from the cat command, meaning the run/secrets/jot file in the container filesystem is empty – no secret to see in there! Even if there is a secrets artifact in our container, Docker is smart enough to not store any sensitive data in the container.

That said, even though this container configuration is secure, it has one shortcoming. It depends on the existence of a file called token1.jwt in the local filesystem when you run the container. If you rename the file, an attempt to restart the container fails. (You can try this yourself by renaming [not deleting!] token1.jwt and running the docker compose command from Step 1 again.)

So we are halfway there: the container uses secrets in a way that protects them from easy compromise, but the secret is still unprotected on the host. You don’t want secrets stored unencrypted in a plain text file. It’s time to bring in a secrets management tool.

Challenge 4: Use a Secrets Manager

A secrets manager helps you manage, retrieve, and rotate secrets throughout their lifecycles. There are a lot of secrets managers to choose from and they all fulfill similar a similar purpose:

  • Store secrets securely
  • Control access
  • Distribute them at run time
  • Enable secret rotation

Your options for secrets management include:

For simplicity, this challenge uses Docker Swarm, but the principles are the same for many secrets managers.

In this challenge, you create a secret in Docker, copy over the secret and API client code, deploy the container, see if you can extract the secret, and rotate the secret.

Configure a Docker Secret

  1. As is tradition by now, change to the apiclient directory:

    cd ~/microservices-march/auth/apiclient

  2. Initialize Docker Swarm:

    docker swarm init
Swarm initialized: current node (t0o4eix09qpxf4ma1rrs9omrm) is now a manager.
...

  3. Create a secret and store it in token1.jwt:

    docker secret create jot ./token1.jwtqe26h73nhb35bak5fr5east27

  4. Display information about the secret. Notice that the secret value (the JWT) is not itself displayed:

    docker secret inspect jot[
  {
    "ID": "qe26h73nhb35bak5fr5east27",
    "Version": {
      "Index": 11
    },
    "CreatedAt": "YYYY-MM-DDThh:mm:ss.msZ",
    "UpdatedAt": "YYYY-MM-DDThh:mm:ss.msZ",
    "Spec": {
      "Name": "jot",
      "Labels": {}
    }
  }
]

Use a Docker Secret

Using the Docker secret in the API client application code is exactly the same as using a locally created secret – you can read it from the /run/secrets/ filesystem. All you need to do is change the secret qualifier in your Docker Compose YAML file.

  1. Take a look at the Docker Compose YAML file. Notice the value true in the external field, indicating we are using a Docker Swarm secret:

    cat docker-compose.secretmgr.yml---
version: "3.9"
secrets:
  jot:
    external: true
services:
  apiclient:
    build: .
    image: apiclient
    extra_hosts:
      - "host.docker.internal:host-gateway"
    secrets:
      - jot

    So, we can expect this Compose file to work with our existing API client application code. Well, almost. While Docker Swarm (or any other container orchestration platform) brings a lot of extra value, it does bring some additional complexity.

    Since docker compose does not work with external secrets, we’re going to have to use some Docker Swarm commands, docker stack deploy in particular. Docker Stack hides the console output, so we have to write the output to a log and then inspect the log.

    To make things easier, we also use a continuous while True loop to keep the container running.

  2. Copy the app for this challenge – the one that uses a secrets manager – to the working directory, overwriting the app.py file from Challenge 3. Displaying the contents of app.py, we see that the code is nearly identical to the code for Challenge 3. The only difference is the addition of the while True loop:

    cp ./app_versions/best_secretmgr.py ./app.pycat ./app.py
...
while True:
    time.sleep(5)
    try:
        with urllib.request.urlopen(req) as response:
            the_page = response.read()
            message = response.getheader("X-MESSAGE")
            print("200 " + message, file=sys.stderr)
    except urllib.error.URLError as e:
        print(str(e.code) + " " + e.msg, file=sys.stderr)

Deploy the Container and Check the Logs

  1. Build the container (in previous challenges Docker Compose took care of this):

    docker build -t apiclient .

  2. Deploy the container:

    docker stack deploy --compose-file docker-compose.secretmgr.yml secretstack
Creating network secretstack_default
Creating service secretstack_apiclient

  3. List the running containers, noting the container ID for secretstack_apiclient (as before, the output is spread across multiple lines for readability).

    docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.RunningFor}}\t{{.Status}}"CONTAINER ID  ...  
20d0c83a8b86  ... 
ad9bdc05b07c  ... 

    ... NAMES                                             ...  
    ... secretstack_apiclient.1.0e9s4mag5tadvxs6op6lk8vmo ...  
    ... exciting_clarke                                   ...                                 

    ... IMAGE              CREATED          STATUS
    ... apiclient:latest   31 seconds ago   Up 30 seconds
    ... apiserver          2 hours ago      Up 2 hours

  4. Display the Docker log file; for <container_ID>, substitute the value from the CONTAINER ID field in the output from the previous step (here, 20d0c83a8b86). The log file shows a series of success messages, because we added the while True loop to the application code. Press Ctrl+c to exit the command.

    docker logs -f <container_ID>200 Success apiKey1
200 Success apiKey1
200 Success apiKey1
200 Success apiKey1
200 Success apiKey1
200 Success apiKey1
...
^c

Try to Access the Secret

We know that no sensitive environment variables are set (but you can always check with the docker inspect command as in Step 2 of Examine the Container in Challenge 2).

From Challenge 3 we also know that /run/secrets/jot file is empty, but you can check:

cd extractdocker export -o api3.tar 
tar --extract --file=api3.tar run/secrets/jot
cat run/secrets/jot

Success! You can’t get the secret from the container, nor read it directly from the Docker secret.

Rotate the Secret

Of course, with the right privileges we can create a service and configure it to read the secret into the log or set it as an environment variable. In addition, you might have noticed that communication between our API client and server is unencrypted (plain text).

So leakage of secrets is still possible with almost any secrets management system. One way to limit the possibility of resulting damage is to rotate (replace) secrets regularly.

With Docker Swarm, you can only delete and then re‑create secrets (Kubernetes allows dynamic update of secrets). You also can’t delete secrets attached to running services.

  1. List the running services:

    docker service lsID             NAME                    MODE         ... 
sl4mvv48vgjz   secretstack_apiclient   replicated   ... 


    ... REPLICAS   IMAGE              PORTS
    ... 1/1        apiclient:latest

  2. Delete the secretstack_apiclient service.

    docker service rm secretstack_apiclient

  3. Delete the secret and re‑create it with a new token:

    docker secret rm jot
docker secret create jot ./token2.jwt

  4. Re‑create the service:

    docker stack deploy --compose-file docker-compose.secretmgr.yml secretstack

  5. Look up the container ID for apiclient (for sample output, see Step 3 in Deploy the Container and Check the Logs):

    docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.RunningFor}}\t{{.Status}}"

  6. Display the Docker log file, which shows a series of success messages. For <container_ID>, substitute the value from the CONTAINER ID field in the output from the previous step. Press Ctrl+c to exit the command.

    docker logs -f <container_ID>200 Success apiKey2
200 Success apiKey2
200 Success apiKey2
200 Success apiKey2
...
^c

See the change from apiKey1 to apiKey2? You’ve rotated the secret.

In this tutorial, the API server still accepts both JWTs, but in a production environment you can deprecate older JWTs by requiring certain values for claims in the JWT or checking the expiration dates of JWTs.

Note also that if you’re using a secrets system that allows your secret to be updated, your code needs to reread the secret frequently so as to pick up new secret values.

Clean Up

To clean up the objects you created in this tutorial:

  1. Delete the secretstack_apiclient service.

    docker service rm secretstack_apiclient

  2. Delete the secret.

    docker secret rm jot

  3. Leave the swarm (assuming you created a swarm just for this tutorial).

    docker swarm leave --force

  4. Kill the running apiserver container.

    docker ps -a | grep "apiserver" | awk {'print $1'} |xargs docker kill

  5. Delete unwanted containers by listing and then deleting them.

    docker ps -a --format "table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.RunningFor}}\t{{.Status}}"docker rm <container_ID>

  6. Delete any unwanted container images by listing and deleting them.

    docker image list   docker image rm <image_ID>

Next Steps

You can use this blog to implement the tutorial in your own environment or try it out in our browser‑based lab (register here). To learn more on the topic of exposing Kubernetes services, follow along with the other activities in Unit 2: Microservices Secrets Management 101.

To learn more about production‑grade JWT authentication with NGINX Plus, check out our documentation and read Authenticating API Clients with JWT and NGINX Plus on our blog.

Read more blog posts about F5 NGINX ›

"This blog post may reference products that are no longer available and/or no longer supported. For the most current information about available F5 NGINX products and solutions, explore our NGINX product family. NGINX is now part of F5. All previous NGINX.com links will redirect to similar NGINX content on F5.com."