The State of Application Strategy 2021: Measuring What Matters

Insights come from analysis of data. Consider that right now, as you read this, your human body is generating about 11 million bits of ‘data’ per second. That’s approximately 1.3 MB, which is just over 1 KB. For comparison, the average log entry for an application transaction is 1 KB. Like a living organism, every action by an application generates data. Requests, responses, errors, warnings, status. So do the services and infrastructure that deliver and secure an application. The amount of telemetry generated by just a single application interaction is more than our brains can take in and process, which scientists believe is about 50-60 bits per second.

But that’s the interesting part. We don’t have to consciously process the data our bodies generate. We have all sorts of internal helpers that process and act on our behalf, automatically. You don’t have to think about breathing or forcing your heart to pump. You don’t just suddenly decide you’re hungry; some internal (hidden) system alerts you to that reality. Every system has its helpers.  

If you think about telemetry, you’ll recall seeing requests from helpers all the time: for diagnostics, for analytics, for performance, for more data. Every technology provider takes advantage of as much data as they can get to improve and optimize their products.

Business shouldn’t be any different. For a digital business, one of its products is the customer experience. You’ll need data to understand how and when to scale and optimize and, of course, to secure it.

Just like the human body, we don’t expect an application to produce and analyze all the data needed to optimize and secure the digital experience. We expect there are helpers that collect, analyze, and instruct systems to do that for the application.

The market agrees that some of the best helpers are application security and delivery technologies.

Telemetry is Important. Period.

Analytics is a rising technology star. It's central to everything from AI to security to managing your app portfolio. Recognizing its importance, we asked a lot of questions this year about data, how it's used, and what the market is missing.

Turns out what it’s missing is, well, a lot. 

Respondents told us they were missing insights that help them identify root causes of performance degradation (49%), incidents that cause outages (51%), and possible attacks (45%). And yet most of these respondents (59%) also told us they have the tools they need to monitor the health and status of their applications.

This disconnect is caused not by a lack of tools or data, but by a lack of analytical processing that goes beyond visualization to offer information and, ultimately, insights. While just about every tool provides a visualization of data points that tell us the state of an application (up, down, slow, fast, etc.), we don’t generally have the tools necessary to provide deeper insights—such as root cause or anomalies that indicate a potential attack.

The importance placed on telemetry was ubiquitous, but we note that its importance was highest for operational service-level objectives (SLO) rather than those used to satisfy business outcomes. 

We suspect the emphasis on traditional metrics are just that: an emphasis on measuring the effectiveness of IT and operations based on individual and, we might add, binary metrics.

Measure what Matters

What we measure must change.

The availability of a given application, for example, may have an impact on the customer experience. But it may not. Modern methodologies like Agile and DevOps and SRE practices are, in part, driven by the expectation that components will fail. The ability to compensate, automatically, is built into modern applications and the technologies that deliver and secure them. But this key principle has not yet been fully adopted by IT and operations and—based on our analysis—the C-suite. People work toward what they’re measured on. If leaders are measuring their organizational effectiveness based on traditional metrics, then those metrics will drive priorities. And that’s what we’re seeing in our research.

That’s troubling given the significant movement to modernize we’re seeing. As modernization efforts proceed apace, the emphasis necessarily shifts from applications to workflows, from user interfaces to user experiences. The best measure of user experience is not binary metrics for availability or performance, but rather business outcomes. Traditional operational metrics—when aggregated and analyzed—can provide the insights necessary meet business outcome expectations. But on their own, traditional metrics are little more than warning alerts that may or may not indicate a real issue.

What’s worse is that many organizations aren’t even monitoring the health and status of the components they use to modernize core business applications. Nearly one-quarter (24%) told us they don’t track SLAs for modern components that create today’s user experiences. These components—usually mobile and container-native apps—are the first impression a user has of a brand. And yet a significant percentage of organizations aren’t trying to understand that experience. A non-trivial percentage (6%) don’t track SLAs at all. For anything.

Anecdotally, I think I’ve used some of those apps—and subsequently deleted them.

Other organizations are tracking component metrics you’d expect:

• 62% response time
• 60% reliability
• 59% uptime

Overall, we learned two things from our research this year: first, telemetry from ‘helpers’—application security and delivery technologies—is a critical component to how IT and the business measure success. Second, organizations recognize the tools they have in place are not discovering the critical insights business and IT needs to deliver extraordinary digital experiences.

Conclusion: Analytics Must Evolve Along with the Business

Digital transformation is a business journey accompanied by a technology evolution. The business and IT must travel the same path forward—toward digitization. To arrive at the destination—at AI-assisted business—requires a partnership between IT and the business. That partnership includes alignment on analytics that shifts strategy to focus on business outcomes instead of operational metrics.

There’s a lot more about analytics and telemetry how organizations are using—and wishing they could use—them in our 2021 official report. Check it out, and don’t forget to check back here as we’re diving into automation next.