10 Questions to Ask a Bot Mitigation Vendor

You suspect that you have a bot problem. Maybe you have increased infrastructure costs due to traffic surges, a high account takeover (ATO) rate, or someone’s cracking your gift cards and scraping your intellectual property. Most likely, if you dig deeper, you will also find a fraud problem, as well as recent falls in brand scores and customer loyalty. You tried to handle it yourself with rate-limiting, IP and geo-blocking, reputation, fingerprinting, CAPTCHA, and Multi-Factor Authentication (MFA), but it became an endless battle managing multiple solutions, trying to stay ahead of attackers, and dealing with frustrated customers who were unable to complete their purchase.

Now you’ve decided to call in professionals to help and get some of your time back. You’ve narrowed it down to a few vendors, and you’re going to ask them some questions. But what questions?

Here are some good ones that can give you an idea if the bot mitigation vendor’s solution is the right fit for your environment.

1. How does the vendor handle attacker retooling?

If the perceived value in your customer accounts is high, attackers likely won’t give up easily—instead they will continuously retool and try again. This is basic attacker economics, which means this is your most important question. When a security countermeasure is put in place, persistent attackers will retool to bypass the mitigation control using various methods, tools, and even AI. Victims of credential stuffing say that fighting bots and automation by themselves is like playing whack-a-mole. You are paying a service to play this game for you, so ask how they handle it.

2. Does the vendor dramatically increase customer friction?

CAPTCHA and MFA dramatically increase friction to your customers. Human failure rates range from 15% to 50% and lead to high cart-abandonment and decreased user satisfaction. And your customer may never come back, even after a single negative user experience.

Honestly, think carefully about vendors who rely on countermeasures known to introduce friction. Not only do these defenses frustrate users, but attackers are often able to get around them anyway. Fraudsters can leverage tools and human labor to solve CAPTCHAs, and can even leverage compromised PII to impersonate account holders in order to move phone lines to accounts under their control in order to complete MFA requests.

3. How does the service deal with false positives?

A false positive for a vendor is when they mark a real human as a bot. A false negative is when they mark a bot as human.

Bot mitigation will have some of both; be suspicious of any vendor who claims otherwise. But a vendor should be very responsive to the problem of false positives; that is, you should be able to contact them, complain, and have the false positive determination addressed.

4. When an attacker bypasses detection, how does the service adapt?

There will be advanced attackers who attempt and even manage to bypass detection, becoming a false negative. A bot mitigation vendor must operate as if a skilled attacker will imminently bypass all countermeasures. When it happens, you may not know about it until you see the side effects (account takeover, fraud, skewed business analytics, etc.). Then you’ll need to contact your vendor and work with them on how to remediate.

How do they handle this process? What is the vendor’s response and turnaround time?

5. How does the vendor handle manual (human-driven) fraud?

If your vendor is particularly adept at keeping out automation (bots), a very, very determined and skilled attacker will input credentials by hand in real browsers in order to bypass anti-automation defenses, potentially leading to Account Takeover (ATO) and Fraud. In fact, a mere $10 in value behind a secure logon may be enough to motivate a professional attacker. Many services do not detect this (since humans are not bots).

The vendor must be able to determine if a human is a trustworthy customer or a fraudster.

Can their service detect malicious intent? Accurate detection can distinguish between a bot, an attacker using sophisticated tools and AI to emulate or exhibit human behavior, and a real customer, and then take the most appropriate action without assisting in the attacker’s reconnaissance efforts or impacting the customer experience. 

6. If one customer gets bypassed, how does the vendor protect that bypass from affecting all other customers?

In many cases, custom detection and mitigation policies should be deployed for every customer. That way, if an attacker retools enough to get around the countermeasures at one site, they can’t automatically use that playbook to get into your site. Each customer should be insulated from a retool against a different customer.

Certain industry verticals such as banking and financial services attract the most motivated and sophisticated cyber criminals. The most effective bot mitigation solutions will detect attackers’ techniques across similar attack profiles and risk surfaces in order to deploy appropriate countermeasures automatically. Additionally, historical fraud records can further train AI models to maximize efficacy.

7. If an attacker bypasses a mitigation, does the service still have visibility into the attack?

It is very common for a service to be blind after an attacker bypasses defenses. If the vendor mitigates on the data they use to detect, then when an attacker bypasses mitigation, you lose the ability to detect. For example, if they block on the IP, when the attacker bypasses the block (distributes globally) the vendor may lose visibility and won’t know how bad the attack really is.

An example of a system that is working correctly is when 10,000 logins come through and they all look okay initially because they have behavioral analytics within the proper range for humans. But later it is determined that all 10,000 had identical behaviors, which means the logins were automated.

The most effective bot mitigation solutions continuously collect and analyze various device, network, environment, and behavioral telemetry signals to maximize visibility and accurately identify anomalies. This, in turn, improves efficacy of closed-loop AI models while providing key insights to the vendor’s Security Operations Center (SOC).

8. How difficult is the solution to deploy and maintain?

Does the user or administrator have to install custom endpoint software, or is protection automatic? If there is no endpoint presence how does the vendor detect rooted mobile devices? How does it detect attacks using the latest security tools and data from the Dark Web? What about APIs?

It helps when your team invents 5 of the 9 relevant features in JavaScript and your solutions run across clouds and architectures, seamlessly deploying customized protections that maximize visibility and security efficacy without introducing friction to the application development lifecycle or customer experience.

9. What types of anomalies does the vendor detect?

Attackers are constantly leveraging bots, automation, and compromised credentials to assist with their efforts, with the endgame being financial gain. Basic mitigations are not enough. For example, attackers re-use IP addresses, but typically only 2.2 times on average. Often, they are only used once per day or once per week! This makes IP blocking largely ineffective.

Automation tools can construct tailored attacks that bypass rate limiting, CAPTCHA solvers and web stack emulation can spoof header and environment checks, scriptable consumer browsers and anti-fingerprinting tools can beat fingerprinting and even behavioral analysis. It truly is an arms race.

Attackers typically invest along four vectors:

• Spoofing network traffic
• Emulating a variety of valid devices and browsers
• Using stolen credentials, PII, and synthetic identities
• Emulating and exhibiting actual human behavior

There are over a hundred client signals besides IP address. A good service will leverage a variety of signals and AI to provide actionable insights and detect anomalous behavior indicative of fraud—including copying and pasting activity, screen toggling, odd screen real estate usage, device affinity, environmental spoofing, and attempts to anonymize identity—rather than relying on IP blocking, signatures, and fingerprinting.

10. How quickly can the vendor make a change?

When the attacker retools to get around current countermeasures, how quickly will the vendor retool? Is it hours, or is it days? Does the vendor charge extra if there is a sophisticated persistent attacker and multiple countermeasures or consultations with the SOC are needed?

There are other questions that are tablestakes for any vendor. Things like deployment models (is there a cloud option?) and cost model (clean traffic or charge by hour?). And, of course, you should compare the service level agreement (SLA) of each vendor. But you were probably going to ask those questions anyway (right?).

Yes, this article is slightly biased, as F5 is the premier application security vendor. But consider the hundreds of customers we’ve talked to who chose us; these are questions they asked, and we hope that they help you, even if you end up choosing a different bot mitigation vendor. Because attackers do not discriminate based on cloud, architecture, CDN, or your security and fraud team org charts. The money is behind your apps and that is where criminals attack.

Next Steps

The world’s most valuable brands, including financial institutions, retailers, airlines, and hotel chains, are already using F5 bot mitigation solutions to protect their apps, customers, and business.

F5 consistently stops more bots and automation than other vendors. This reduces fraud and operational complexity while increasing revenue and improving the customer experience.

Let us know if you’d like to see for yourself.

Portions of this post are based on previously published content that has been updated to reflect F5's current offerings.