Much of our threat research is focused on analyzing quantitative threat data—the larger the sample size, the better. However, the critical piece of information about a specific attack, the thing that differentiates it from the attacks that happened the day before or after, often lies in minute detail that is rarely captured in a large dataset. For that reason, it can be useful to balance quantitative threat research with qualitative, anecdotal accounts of particularly instructive attacks.
That’s what we’re going to do today. One of the customer-facing security operations centers (SOCs) at F5 recently observed an attack taking place against a customer. In the spirit of learning from setbacks and adapting to the latest attack methods, we feel this data breach is a particularly useful event to analyze in detail because the attackers’ activities revealed interesting things about both their strategy and tactics. While we can’t reveal too much about the attack or the victims for confidentiality reasons, we can still share enough to draw some conclusions about the threat landscape.
The Attack: Basic Facts
Like many breaches we’ve covered in the past, this attack began with the exploit of a PHP vulnerability on a Web-facing server. This foothold allowed them to exploit the PwnKit vulnerability to achieve remote code execution inside the /tmp/ folder on the server, which they used to write a bash shell into the directory.1 From there the attacker moved laterally to a database containing sensitive information. The database contained user credentials, but more importantly, it also contained an unsecured key for an internal east-west API. This key would prove crucial to the success of the attack because it allowed the attackers to bypass rate limits designed to detect large-scale data transfers and exfiltration.
At that point, the attackers prepared a single user account as a staging point to collect data for exfiltration. Because they had access to the database containing user data and credentials, they were able to bypass the multifactor authentication (MFA) protecting that user account by changing the phone number designated for MFA to a number they controlled.
Once they had prepared this staging point for exfiltration, the attackers spent several days slowly transferring sensitive data into that account. When the data was in place, the attackers socially engineered a distraction for the customer's in-house security team. Once security staff was distracted, the attackers exfiltrated all of the staged data in one big move. They then wiped the compromised server on their way out of the environment to hamper forensic investigation.
Learning From the Attack
Even though this is just one attack, the visibility that the F5 SOC team had into attacker behaviors means that we have a rare bit of detail about both what they were trying to accomplish, and how they went about doing it. This lets us form some hypotheses about how this attack differs from a “typical” attack.
Sophistication Is About More Than Just 0days
The fact that so many different tactics and techniques came together successfully for the attacker suggests that the threat actor was more sophisticated than many. This illustrates a bit of a conundrum in terms of how we talk about sophisticated threats. In much of the discussion of the threat landscape, the differentiator between these higher-tier threats and more run-of-the-mill cybercrime actors is often treated as no more than access to significant zero-day exploits and quickly achieving persistence. Whether this threat actor “counts” as a sophisticated actor or not (which often boils down to whether they are backed, overtly or covertly, by the governments of a handful of nation-states), we would suggest that this attack contains two other characteristics that should be used to differentiate sophisticated attacks from more routine activity.
First Aspect: Coordination & Planning
The first aspect is the degree of coordination, organization, and situational awareness that the attacker displayed. The attacker had clearly done thorough reconnaissance on their target before beginning the attack and already knew what their plan was for the first several steps of the attack. Similarly, the attacker was able to coordinate all of their actions—even the unplanned ones—with an eye toward maintaining stealth until exactly the right moment came to move fast and loud. This kind of higher awareness of their own conspicuousness allowed them to segment their entire attack chain into high and low visibility phases and plan from there.
Second Aspect: Flexibility & Opportunism
The other notable thing about this attack is the actors’ ability to take advantage of things that they could not have foreseen—most notably the hardcoded API key that allowed them to stage so much sensitive data for exfiltration. Of course, hardcoded credentials of any type are always interesting to attackers, but the specific detail that the API allowed them to avoid rate limiting ended up being a key part of their exfiltration plan. The ability to capitalize on this in the moment goes beyond basic “Living Off of the Land” techniques and illustrates both broad and deep knowledge of modern enterprise architectures.
On the surface, these two aspects look somewhat opposed, but this is really what people mean when they talk about making your own luck. The degree of planning and coordination the attackers brought with them also set them up to take advantage of unforeseen opportunities. In short, these attackers made their own luck, and this, as much as any specific tactic, technique or procedure, should be understood as a significant differentiator of sophistication.
Internal APIs Can Be Just as Dangerous as Public APIs
As APIs have grown into one of the most common and indispensable components of modern architectures, the security industry has also recognized that APIs can dramatically increase attack surface. Many of the most notable API security incidents, however, revolve around technically simple attacks against public facing, poorly-secured API endpoints, such as Identity Provider (IDP) APIs.2 In this case, an internal API was an important part of the attackers’ collection and exfiltration tactics. While the risk of public APIs is more obvious, practitioners and enterprise architects should also recognize the opportunities that internal APIs offer to attackers trying to live off the land. This is one area in which a Zero Trust approach could help mitigate the impact of an API compromise even if an attacker has legitimate credentials.
Real-Time Visibility Is Increasingly Important
Simultaneous shifts in enterprise architectures, application architectures, development styles, and business models have all combined over the last five or so years to increase attack surface and challenge situational awareness for most organizations. It was not for nothing that we subtitled the 2019 Application Protection Report “The Virtue of Visibility.” The only reason that we even have the information to publish this analysis is because of F5’s capabilities to detect complex combinations of attacker behaviors at multiple levels. No other detective controls present in the environment of F5’s customer managed to flag the malicious activity with nearly as much precision.
This tells us that a new generation of controls offering enhanced visibility might become increasingly indispensable as attackers take advantage of the expanded opportunities our new arrangements seem to offer them.
Use MFA But Understand Its Limitations
While the use of MFA is becoming commonplace, many organizations still rely on its weakest implementation, SMS-based one-time passwords. Numerous methods exist to bypass SMS-based MFA including SIM swapping, mobile malware, and, as in this attack chain, simply changing the assigned phone number for the user being attacked. From mobile malware, such as Malibot, to real-time phishing proxies, F5 Labs has been tracking the growing number of threat groups which are capable of bypassing all forms of MFA. While no solution is perfect, SMS-based MFA is the least perfect of the MFA options, and NIST has long recommended against its use, so organizations that do implement it should be clear about the ease with which determined attackers are able to bypass it.
Tune Your Alerting Systems
Ultimately, this attack was successful despite warnings from the F5 SOC engineers which could have stopped it. Alert fatigue is a real problem and there are many examples of successful breaches occurring simply because alerts were ignored by the engineers managing the security controls. This is why it’s increasingly important to form a good relationship with managed security services and take their recommendations seriously.
Recommendations
Recommending mitigations for the techniques used in this attack is tricky. On the one hand, all of the observed techniques in this attack chain are theoretically preventable. On the other hand, this threat actor was clearly motivated and possessed many tricks up their sleeve, so it is possible the attack might have succeeded via another vector. Nevertheless, we can make a few observations about practices that would have contained the risk we observed in this attack:
- Technical/Preventative: Patch web vulnerabilities as urgently as possible.
- Technical/Detective: Employ a web application firewall to detect and mitigate web exploits.
- Administrative/Preventative: Establish developer and administrator policies to prevent and detect hardcoded credentials in code or databases.
- Technical/Detective: Scan development and production environments for sensitive information such as credentials, personally identifiable information, etc.
- Technical/Preventative: Consider multi-factor authentication solutions and understand their limitations, particularly SMS based one-time passwords.
- Technical/Preventative: Employ access control for API connections to prevent their use in collection and exfiltration tactics.
- Technical/Preventative: Use a Zero Trust architecture to prevent lateral movement, discovery and collection during a network intrusion.
- Technical/Detective: Employ leading-edge threat detection tools for enhanced visibility.
