Introduction
The 2025 Advanced Persistent Bot (APB) Report covers all bot types, across all industries, at a high level. But web scrapers, in particular, are a persistent threat, growing even more so with the seemingly infinite appetite of generative AI platforms. Scrapers extract large amounts of data and content from their targets, and usually well beyond the volume that data and content owners intend to provide. Bots are responsible on average for 10.2% of all traffic across all platforms. However, specific industries are much harder hit.
Fashion was the worst hit industry, seeing 53.23% of all web traffic coming from scrapers. Insurance saw the most targeting from the highest sophistication of scrapers with 59.83% of all bot traffic for web coming from ‘advanced’ scrapers. Mobile API traffic saw much lower volume than web traffic. For mobile APIs, the worst hit industry was in Hospitality with 4.53% of all traffic coming from scrapers. Banks and credit unions saw the largest proportion of ‘advanced’ scrapers (99.24%) targeting their mobile APIs.
Scraper Bot Behavior in Response to Bot Mitigation
Scrapers are mostly active on web and not on mobile APIs. Average automation on web is 17.61%, compared to 1.07% for mobile API. However, it’s important to understand how to interpret our statistics, as we are presenting data aggregated from F5 customers with bot mitigations in place. If you don’t have mitigations in place already then you will be likely experiencing higher scraper bot volume than we present here, and likely bot scrapers of lower sophistication. Viewed through the lens of an adversarial dynamic this makes intuitive sense.
Here’s how bot mitigations drive bot operators to alter their behavior and what it means for our data for your industry. Bot operators may:
- Go away – bot operators turn their attention to other businesses without bot controls in place. This leads conservative scraper volume in our post-mitigation numbers, and hence why our scraper volumes should be viewed as best-case volume when mitigations are not in place.
- Increase in sophistication – bot operators “retool”, alter the behavior of their bots, and attempt to circumvent the new controls. This can lead to generous scraper sophistication in our post-mitigation data. Therefore, without mitigations in place you can expect lower sophistication of bot scrapers.
- Make no changes – bot operators don’t respond to bot mitigations. This leads to no difference between pre-mitigation and post-mitigation numbers, as the scrapers continue scraping activity even though the scraping fails. The reason we mention this is because this happens in certain industries more than others and can cause confusion when interpreting our data. Certain industries will see bot operators employ broad targeting. When targeting is broad it doesn’t always make economic sense for a bot operator to invest in improvements. This explains why we see high variation by industry in scraper volume in our post-mitigation numbers.
Top Scraped Industries
Many industries face a high degree of targeting by scraper bots, some more than others. Here, we break down which sectors are most likely to have content scraped via their web applications or mobile APIs.
Web
We see high volumes of web scraping automation for Fashion (53.23%), Hospitality (49.32%), and Healthcare (34.47%) despite bot mitigations blocking their activity (see Figure 1). This is explained by the economics of scraping these industries. These industries are crowded markets with many targets. As such, bot operators are scraping the market as a whole. Running bot scrapers is relatively inexpensive, so often it’s not worth investing the effort to stop scraping a specific target. This is why Fashion, Hospitality, and Healthcare see such a high volume of activity from scrapers.
We see lower volumes of web scraping automation for Airline (10.76%), Bank/Credit Union (7.73%), and State/Local Government (7.72%). Here the economics have shifted, this time due to the value of scraping these industries. In these industries bot operators are targeting selectively, as individual targets are not fungible to the degree they are in Fashion, Hospitality, and Healthcare.
Lastly, we see the smallest volumes of web scraping automation for Entertainment (1.88%), Gaming (1.32%), Insurance (0.10%), and Telecom (0.08%). Historically these industries saw high levels of scraping. However, since these industries are far more mature in their bot mitigations, having been the target for comparatively much longer than other industries. This explains the overall lower volume of bot scraping attempts for Entertainment, Gaming, Insurance, and Telecom industries.
Mobile API Scraping Automation by Industry
On mobile API, we see the highest volumes of scraping in Hospitality (4.53%) followed by Bank/Credit Union (4.17%), as shown in Figure 2. Scrapers on mobile API banks and credit unions are mainly targeting credit unions, and are searching for branch and ATM locations, interest rates, card fees and pricing information, as well utilizing the member search feature of credit unions.
Sophistication
As with all threat actors, bot operators will rarely spend time or money on developing advanced bot capabilities unless they feel they must, so it is quite fascinating to see the huge range in bot capabilities targeting various industries. We refer to bots as having either ‘basic’, ‘intermediate’, or ‘advanced’ sophistication.
Web
The sophistication levels of web scrapers vary widely by industry. The highest proportions of basic scrapers were in Healthcare (98.30%), Fashion (94.09%) and Telecoms (80.96%), as represented by the darkest shades in the bars show in Figure 3. Scrapers in these industries are simple HTTP scrapers, perhaps using the curl command line, or simple Python scripts. Overall, 54.05% of all web scrapers were basic level sophistication. Only two industries had the majority of web scrapers exhibiting Advanced sophistication, these are Insurance (59.83%) and Airlines (50.46%). It is not surprising as these industries are targeted by well-resourced actors doing insurance rate scraping, premium comparisons and network scraping in the insurance space, and airline fare, availability and seat scraping in the airline space.
Mobile API Scraper Sophistication
Mobile API scraper sophistication differs significantly from that of web scrapers. Mobile API scrapers mainly exhibited intermediate levels of sophistication. We found that 65.46% (2 in 3) mobile API scraper transactions were of intermediate sophistication. The only industry to see large proportions of advanced scrapers was Bank/Credit Union (99.24%). See Figure 4.
Conclusion
Scrapers are a persistent threat – to some industries more than others. We’ve examined and explained industry variations in scraper volume and sophistication for both web and mobile API. For insight into the direct and indirect cost of scrapers, see the F5 Labs article “What are Scrapers and Why Should You Care”. For further reading, F5 Labs also has a nuanced guide to identifying and stopping scrapers, however, be advised of the risks in developing in-house solutions. To take action, consult the recommendations of the 2025 Advanced Persistent Bot Report.
