Create a Secure Multi-Tenant Architecture with F5 VELOS

Businesses are focusing on containing costs in their enterprise data centers while supporting increased demands on content delivery. As these organizations seek to maximize their utilization rates of existing infrastructure, ensuring data security and compliance has become a top priority. Because many organizations have workloads distributed across on-prem, colocation, and multi-cloud deployments, they need a consistent data-security strategy across all their deployments. F5’s new modern architecture enables organizations to accelerate digital transformation plans by deploying F5 software in a modular fashion, on both private and public clouds, while preserving their existing hardware acceleration capabilities. F5's microservices-based infrastructure supports application delivery and multi-tenancy functionality with comprehensive security isolating tenants – all a cornerstone of F5’s Adaptive Apps vision.

Data-security issues impact one in three service providers. Given the rise in data breaches and fines for regulatory and compliance issues, managed service providers want to assure downstream customers that their network traffic cannot be seen or manipulated by other customers hosted on the same physical device.

F5’s modern architecture combines the high-performance characteristics that chief information officers (CIOs) require with the robust, high-security posture that chief (information) security officers (CSOs/CISOs) demand.

Previous-generation systems like VIPRION offered virtualization solutions through vCMP technology. vCMP had two components that provided multitenancy: the vCMP host layer and the vCMP guest layer. The vCMP host functionality was essentially a custom hypervisor that allowed F5 platforms to run virtual instances of BIG-IP, which were called guests.

VELOS provides a model similar to vCMP from a configuration and deployment point of view, but with different technology. VELOS uses the terms “tenants” or “tenancy” to describe the virtualization capabilities within the chassis. The F5OS platform layer in VELOS is not a hypervisor, but a real microservices layer with an underlying Kubernetes framework for management. F5OS implements multi tenancy by using KubeVirt technology to allow BIG-IP instances to run as virtual machines (VMs) on top of the microservices layer. This enables customers to migrate existing BIG-IP instances, or guests, to VELOS tenants without changing how they manage their tenants.

In the future, the VELOS architecture will also support the next generation of BIG-IP software. Instead of running BIG-IP instances as a VM on top of a containerized environment, in the future BIG-IP will run as a collection of containers within the native container environment.  Supported tenants on a VELOS chassis can be BIG-IP with TMOS and the modern BIG-IP software in containers on F5OS in the future for seamless testing and version migration on the same platform.

Tenant configuration in VELOS is almost identical to provisioning a vCMP guest. The administrator assigns the tenant a name, selects a software version for it to run, and assigns vCPU and memory resources. VELOS tenants use dedicated CPU and memory resources, just like vCMP guests.

VELOS offers an additional layer of isolation through a chassis partitioning feature that allows administrators to group and isolate blades from each other. Each blade can be its own isolated chassis partition, or it can be grouped with other blades to form larger chassis partitions up to the maximum number of blades within the chassis. Chassis partitioning provides an additional layer of isolation that includes the physical networking. With vCMP, the physical networking at the host layer was accessible by all guests, although it could be restricted by VLAN membership. With VELOS multitenancy and chassis partitions, tenants within a chassis partition only have access to the physical networking within the chassis partition they are hosted on. They are not allowed to access networking within other chassis partitions. They are still restricted by VLANs, just like vCMP, but the chassis partition creates further separation at lower layers.  VELOS further isolates chassis partitions by segregating admin access so that each chassis partition manages its own user access and authentication.

Chassis administrators can access the out-of-band management interfaces of the system controllers. They can also configure chassis partitions and assign user privileges to access these partitions.

Chassis partition administrators configure the in-band networking infrastructure for the blades in the partition, which will include interfaces, link aggregation groups, and VLANs. They can also deploy and manage tenant lifecycles within their assigned chassis partition. A chassis partition administrator will not have access to other chassis partitions in the system, so it provides secure isolation not only at the tenant layer, but also at the lower networking layers.

Tenant administrators are responsible for the configuration of the services within the tenant. Tenant managers have access to all the management functionality exposed by a BIG-IP tenant. This is similar to access to a vCMP guest. It is independent of the lower platform layer access and is controlled within the TMOS instance (or tenant).

Microservices-based modern architecture will help organizations that want to digitally transform their data centers. The ability to innovate and react quickly is the goal of digital transformation and data center modernization. With more service providers and enterprise data center operators deploying workloads across on-prem, colocation, and multi-cloud models to meet expanding app delivery requirements, improving utilization rates and data security are becoming increasingly important. F5’s microservices-based architecture offers enterprises better CapEx utilization while ensuring compliance with data-security standards.

Slower deployment times, capacity expansion constraints, and data-security breaches are the top problems enterprises face. There is steady investment in automation and remote monitoring technology to reduce human errors.

For service providers, 5G readiness and improving subscriber and network performance are top priorities. In addition, they’ll need resource scalability to meet the increasing load, now and during the 3G or 4G LTE migration to the 5G Edge, while ensuring application and resource isolation.

F5 developed the VELOS chassis system to help enterprises improve application performance, fault tolerance, and API-management capabilities.

F5OS, along with the VELOS platform, offers better resource utilization, automation capability, and defense-in-depth security. To enhance resource utilization, VELOS supports multiple clusters of tenants on a single device. New chassis partitioning methodology, along with granular resource allocation, provides a simpler operational model. API-first architecture helps customers automate their BIG-IP system deployment, easing application management and automation activities. VELOS supports multiple layers of application security to limit the scope of an exploit. VELOS also supports a combination of physical and software-based defense-in-depth security mechanisms to protect processes, secure access control, and isolate networks.

Managed service providers have also found that the security of tenants means their downstream customers can independently manage the isolated segments of their own services. For example, one downstream customer may run F5 Advanced Web Application FirewallTM (WAF) or BIG-IP® Advanced Firewall ManagerTM (AFM) from another tenant instance running in the same partition to secure its application, while another customer may be running F5 BIG-IP Access Policy Manager (APM) to provide access control and authentication to its network services.

Security is built into every aspect of VELOS. During VELOS design and development, the F5 product development and security teams examined and performed threat modeling on all the attack surfaces—the most comprehensive security assessment in F5 history.

Chassis partitioning is one of the multitenancy solutions that F5 offers. Chassis partitioning allows customers to create partitions at individual blade levels, grouping the blades together to offer a single managed entity. Each chassis partition has its own management stack and data network connectivity, thus isolating tenants hosted on each of these different partitions.

The F5OS platform layer provides isolation between the applications running within the chassis partition and the underlying hardware by leveraging security context constraint (SCC) profiles and secure computing (seccomp) mode. In addition, the default Security-Enhanced Linux (SELinux) settings will restrict tenant service access to host resources. Authentication and user management restrict application access by creating separate chassis-administrator, partition-administrator, and tenant-administrator roles. The host layer built on a read-only file system further prevents the breakout of any tenants.

Trusted Platform Module (TPM) (ISO/IEC 11889) is an international standard for a secure crypto processor, a dedicated microcontroller designed to secure hardware through integrated cryptographic keys. F5 implements the TPM chain of custody and attestation using the TPM 2.0 chipset, Linux Trusted Boot (tboot), and Intel TXT technology. The TPM chip performs certain measurements each time the system starts. These measurements include taking hashes of most of the BIOS code, BIOS settings, TPM settings, tboot, Linux initial RAM disk (initrd), and Linux kernel (Initial VELOS release only validates BIOS), so that alternative versions of the measured modules cannot be easily produced and so that the hashes lead to identical measurements. You can use these measurements to validate against known good values.

Both system controllers, as well as all the blades (BX110), have a TPM 2.0 chipset. For the initial VELOS release, local attestation is done automatically at boot time and can be displayed in the command-line interface (CLI). In the future, F5 will add remote attestation and attestation for the Linux kernel and initrd.

Administrators can further lock down the VELOS system by enabling appliance mode. Appliance mode is a security model designed specifically for federal and financial security requirements. Appliance mode removes access to the underlying system shell, making script execution much more difficult. Access to the underlying system root account is also removed to force all users through the authentication system.  In VELOS, appliance mode can be enabled for the F5OS platform layer and also within each tenant that is provisioned.  An administrator can enable appliance mode at the system controller level, for each chassis partition, and for each tenant. This mode is controlled by configuration options at each level rather than by a license, as is the case in some VIPRION environments.

To review, the most prominent platform security features are:

• Separation and segregation within the chassis using chassis partitions
• Chassis partition isolation, restricting access to a specific group of users
• Hardware security through Trusted Platform Module (TPM)
• Appliance mode, which prevents script execution and root exploit
• Signature verification, allowing only F5 signed software images

These features secure the platform layer against outside threat vectors. The tenants themselves have their own security subsystems.

With multitenancy functionality, providing isolation among tenants becomes a top concern. Service providers and enterprise customers alike, that host multiple tenants on the same device, need isolation between resources owned by different tenants. F5 ensures tenant-to-tenant security by providing multiple layers of security configurations.

Only F5 trusted tenant services like BIG-IP Local Traffic ManagerTM (LTM), BIG-IP DNSTM, and BIG-IP Advanced WAF may be deployed, minimizing the threat surface. The signature verification helps to authorize the download and installation of the F5 services image. If the signature verification fails, the software installation is terminated, eliminating the risk of malicious activity trying to misconfigure resources.

Isolation between the tenants and host layer resources like process, network, and filesystem, is provided by leveraging various security mechanisms such as SCC profiles and seccomp mode. Tenants have unique tenant IDs for traffic isolation. Tenant addressing, IP table configurations, and the isolation of broadcast domains work together to enforce tenant isolation.

Each tenant offers a role-based access control (RBAC) system to control user access to keys, applications, security policies, audit logs, firewall rules, and more. This means that specific tenants can be assigned to separate downstream customers or business units whose credentials remain separate as well. When tenants are deployed in this way, a compromised user account will remain isolated to a single, specific tenant. Each tenant runs an instance of the TMOS operating system on top of a containerized environment, thus putting it ahead of the vCMP guest in terms of security. Key points include:

• Signature verification allows only F5 signed software images
• MACVLAN interfaces encapsulate inter-blade tenant traffic, restricting traffic visibility between different tenants on the same blade
• Distinct access to partition-administrator and tenant-administrator via platform layer isolating tenants
• Seccomp container security
• Enforcement of SELinux policies to all application services

These security controls combine to provide multiple layers of tenant security.

The new F5OS platform layer is completely isolated from in-band traffic networking and VLANs. It is purposely isolated so that it is only accessible via the out-of-band management network. In fact, there are no in-band IP addresses assigned to either the system controllers or the chassis partitions; only tenants will have in-band management IP addresses and access.

This allows customers to run a secure, locked-down out-of-band management network where access is tightly restricted. The diagram below shows the out-of-band management access entering the chassis through the system controllers, where they provide connectivity to the chassis partitions and tenants. That out-of-band network is then extended within the chassis for management purposes. Note all in-band addressing is on the tenants themselves and not on the F5OS platform layer.

Each chassis partition is a unique entity that has its own set of (local/remote) users and authentication. It is managed via a dedicated out-of-band IP address with its own CLI, GUI, and API access. A chassis partition can be dedicated to a specific group, and members of that group will only be able to access their partition. They will not be able to access other chassis partitions in the system. This is a level of isolation that VIPRION did not have. Below are some examples. You can set up service chaining between different chassis partitions, but they are so isolated within the chassis that you will need to provide external connectivity to link them together.

In addition to management access being completely isolated and unique, in-band networking is configured and completely contained within the chassis partition. Each chassis partition will have its own set of networking components such as PortGroups, VLANs, Link Aggregation Groups (LAGs), and interfaces. This means that networking within one chassis partition is not accessible or viewable from another chassis partition.

Isolation at the network level is also enforced via the centralized switch fabrics that reside in the dual system controllers. In the VELOS system, each blade has multiple connections into the centralized switch fabrics for redundancy and added bandwidth. Each BX110 blade has 2 100Gb backplane connections (one to each system controller), that are bonded together in a LAG. This star-wired topology provides fast and reliable backplane connections between all the blades and also allows for complete isolation at the networking layer.

When chassis partitions are created the administrator will assign one or more blades, which are then isolated from all other blades in the chassis. The centralized switch fabrics are automatically configured with port-based VLANs and VLAN tagging to enforce isolation between chassis partitions. The diagram below provides a visual of how this is enforced.

To illustrate the point of how isolated chassis partitions are, the diagram below shows two VELOS chassis with multiple chassis partitions in each. Because there is no sharing of in-band network resources, each chassis partition must have its own network connectivity to the in-band networks and for high-availability interconnects between the two chassis. There is no way to access interfaces, VLANs, or LAGs between chassis partitions.

As organizations adopt microservices-based application deployment strategies for better resource utilization and end-to-end automation and orchestration, they will have to evaluate how those strategies mix with multitenancy requirements as well as how to maintain total security. F5's modern architecture offers a unique, high-performance, microservices-based solution that meets the needs of both security and multitenancy.

F5 understands the significance of security in a microservices-based environment. F5's extensive threat model assessments on the platform and network connectivity have resulted in a security model built on a defense-in-depth strategy and a proactive security posture. VELOS combines platform layer isolation, chassis partitioning, and centralized switch fabric methodologies to offer isolation between applications in a multitenant environment.