Why ML is mandatory for effective API security

Over the past decade, the quest for faster development cycles, high availability, selective on-demand scaling, and decoupling in general has steered technology-driven organizations to the microservices architecture. As a result, many secure function calls in monolithic software turned into API calls between microservices over non-secure networks. That is, although this transition solved many operational and business challenges, it gave birth to some new security challenges.

The security situation is aggravated with the proliferation of open source software, containerization, and Infrastructure-as-Code (IaC). Now that onboarding or modifying a microservice could be a 15-min task, security teams may not have a timely opportunity to analyze the security of the microservices properly. Introduction of a comprehensive security assessment as a gating phase in deployment pipelines almost always receives an unfavorable reaction from the DevOps team. Even if one was successful at efficiently gating the deployment, many security issues - especially runtime attacks - cannot be identified at the time of deployment.

Security is no longer the same

While developers and DevOps teams have reasons to warmly embrace the transition to microservices, security teams increasingly struggle to keep up with a growing volume of APIs, changing API definitions and behaviors, lack of API documentation (especially in open source components), diverse protocols, and payload structures. Now that many internal function calls between modules have become API calls between microservices, the attack surface has exploded and managing security has become overwhelming very quickly.

Even at medium scale in a microservices environment, defining and maintaining a list of normal and acceptable behavior for each API endpoint before any microservice is deployed or updated is very challenging, if not impossible. This also includes maintaining an authorization policy for each API endpoint. Additions of new microservices, removal of deprecated ones, and seasonality in traffic are the predictable reasons why the definition of “normal and acceptable” keeps changing. There are also not-so-predictable reasons like intermittent network outages, DDoS attacks, upstream outages, and security exploits which can also play a major role.

Given all the factors to be considered for proper security, a large amount of data needs to be collected, correlated, processed and learned in near real-time to make reasonable and timely security decisions. This requirement renders traditional solutions and controls inadequate. Thankfully, advancements in machine learning (ML) and the maturity of the learning models over the past decade provide a very powerful platform to tackle these new security challenges.

ML to the rescue!

The power of ML allows us to rethink the way we define, refine and implement security controls. It allows us to cope with the dynamic nature of API interactions and still provide effective API security. We will now discuss the three main steps in achieving a comprehensive and effective ML-based solution.

1. Let ML Discover the Scope...Don’t Feed It

As discussed earlier, the dynamic nature and sheer volume of APIs make it impractical for developers or security teams to meaningfully define and feed API definitions to systems securing the API endpoints. With ML-based models, it is possible to input some offline training data and let the system autonomously discover, identify, classify, account for, and allow/deny API calls based on the API endpoint (say URL path in case of HTTP), caller, payload, etc. For example, API discovery is accomplished using an ML-based URL graph and component classification can be done using deep learning. This allows the system to build, grow, and learn the scope and nature of API communication over time.

A well-calibrated system for such discovery of scope can replace days, if not weeks, worth of human efforts for each updated deployment within hours.

2. Learn, Adjust, Repeat

This is the core part of any ML-based security solution. Good solutions enable security controls to observe the behavior of the system, build behavior profiles, and adjust the current model (built using the past observations). This is a continuous process, and a cleanly implemented solution enables the controls to be applied in a meaningful way and a timely fashion. For example, if past profiles have shown a particular sequence of APIs calls to be a normal flow, an updated build of application that implements a couple of extra calls in the flow can be learned and added in the profile in a matter of hours.

This is where the argument for ML-based solutions is very easy to understand since the behavior profiling and calibration is as real-time as practically feasible.

3. Detect

The first two steps allow us to learn and keep up-to-date with what should be considered “normal and acceptable” behaviors in the system. The final step is to detect and call out the behavior patterns that fall outside of that range. For example, if after learning the normal behavior of hundreds of users the system notices a sudden flurry of login or data downloading attempts from a particular user, or multiple users attempting login from the same network address in a short amount of time, these behaviors can be quickly called out as anomalies. API anomaly detection can be done using sequential unsupervised deep learning.

It is important to note here that the feedback delay is a very critical part of the entire learning cycle to not allow malicious actors to adjust their behavior and “fly under the radar.”

Figure 2 shows a sequence of API calls that is profiled to be the normal and acceptable by the system.

Figure 3 shows anomalous sequences of API calls that would be detected and flagged by the system.

It is important to note here that the feedback delay is a very critical part of the entire learning cycle to not allow malicious actors to adjust their behavior and “fly under the radar.”

Once these solutions are implemented and start producing results, one can leverage them to enhance existing controls such as web application firewalls (WAFs) for HTTP-based traffic.

Wrapping up

ML is powerful and ML-based solutions have quickly become an essential part of overall security in microservices-based environments. At Volterra, we recognized very early the need for ML for our internal operations, as well as providing it as a key capability for our customers as they transition to modern applications effectively, securely and efficiently.