Now is the Time to Reevaluate Your Bot Countermeasures
If you, as a security professional, think that you have solved your bot mitigation problem, think again.
Bots cause tremendous financial pain: credential stuffing bots lead to account takeover, unauthorized reseller bots make a fiasco of product launches and limited time offers, scraper bots slow performance and raise infrastructure costs, gift card-cracking bots drain balances and infuriate customers. Not only is the impact large, the likelihood of an attack for businesses online is nearly 100 percent because the attacks are so lucrative for criminals. Given the high impact and high probability, you clearly need an effective security strategy. Yet, as recent headlines demonstrate, attackers continue to develop bypasses for commonly deployed bot defenses, making this the right time to reevaluate the countermeasures you have in place against bots.
On January 24, 2023, Joe Berchtold, the President of Ticketmaster’s parent company, Live Nation, testified before the U.S. Senate Judiciary Committee that reseller bots were responsible for its mishandling of ticket sales for Taylor Swift’s upcoming tour. “This is what led to a terrible consumer experience, which we deeply regret,” Berchtold told the senators.
Also in January 2023, CNET reported that thousands of people who use Norton password manager received notices that an unauthorized party may have gained access to their personal information along with the passwords stored in their vaults. Gen Digital, Norton’s parent company, attributed the security incident to a bot-driven credential stuffing attack that it detected when its IDS (Intrusion Detection Systems) system flagged an unusually high number of failed logins.
Ticketmaster had invested in mitigating bots, and we can assume that Gen Digital had protections in place as well. Yet bots still caused security damage, impacting availability and confidentiality, beside generating a lot of bad press. This raises an obvious question: While anti-bot solutions have been available for years, why do so many bot defenses fail to mitigate malicious bots?
For organizations that still rely on CAPTCHA, the answer is obvious. Other than annoying real customers and reducing conversion rates for e-commerce, CAPTCHA does not work. Just do a web search on CAPTCHA solving services and you will find at least a dozen competing on price and speed. The creator of one open source library has made it his mission to make it trivially easy to bypass CAPTCHA:
“CAPTCHAs in their current form have failed. They’re a much bigger obstacle and annoyance to humans than to robots, which renders them useless. My anarchist contribution to this discussion is to demonstrate this absurdity, with a plugin for robots with which a single line of code is all it takes to bypass reCAPTCHAs on any site.”
For organizations that rely on WAF-based IP deny lists to mitigate bots, the task is equally hopeless. There are services available that offer bot creators tens of millions of residential IP addresses intended to bypass bot detection. These services are advertised as rotating residential proxies; the bot sends http requests to the proxy, much like many corporate browsers send requests through forward proxies, and the service continuously rotates the public IP address used to send the request to the website or API. In the past, bots typically used data center proxies, often from cloud services, which are well known and easy to identify. These new proxy services, however, use residential IP addresses from the same geographic area as your customers. Because each IP address could simultaneously represent either a bot or a valid customer due to NATing by ISPs, it is not feasible to block all these IP addresses without turning away your real customers.
Going one step further, new commercial services such as ZenRows, ScrapFly, and ScrapingBee make web scraping as easy as calling an API. The services take on the full responsibility for bypassing bot defenses on your behalf. While these API-based services focus exclusively on scraping, which is legal in the United States and the European Union, criminals have access to similar services on the dark web that perform more nefarious bot attacks such as credential stuffing.
In addition to commercial services, several open source projects are tackling bot bypass. A stealth plugin for Puppeteer, a popular node.js automation and testing tool, enables Puppeteer to bypass detection. An active group of developers maintain the project on GitHub and issue updates whenever a bot defense is known to detect it. According to its documentation, “As this cat & mouse game is in its infancy and fast-paced the plugin is kept as flexible as possible, to support quick testing and iterations.” A similar library, undetected-chromedriver, caters to Python developers. In the spirit of open source, the purpose of these projects is to keep the web open for developers to run automation against apps. Unfortunately, the capability is easily exploitable by criminals.
With more organizations involved in creating open source projects and commercial services, the developers involved in bypassing bot defenses learn and share information. These instructions walk readers through the steps for de-obfuscating the JavaScript of detection tools and deciphering how these tools package their data for transport to their detection service. By reverse engineering the client-side code, these developers pick out how detection tools work. For example, a developer for ZenRows shares what he learned about window sizing and the way bot detection services catch inconsistencies:
“In the sensor data example image, we can see that it sends window size. Most data points are related: actual screen, available, inner, and outer sizes. Inner, for example, should never be bigger than outer. Random values will not work here. You’d need a set of actual sizes.”
With the severity of the threat in mind, it is clearly time to reevaluate your countermeasures for bot mitigation. As the leader in bot detection efficacy, F5 can help. If you want to understand the true state of your bot mitigation efficacy, what bots you are missing, and how much it is costing your business, F5 offers a free threat assessment and bot business impact consultation.