I want to start with a number. Ninety-eight percent. That is the patch compliance rate a team I worked with was proudly reporting to their leadership every quarter. Ninety-eight percent of critical vulnerabilities patched within the thirty-day SLA window. It looked great on a slide. It felt like evidence of a mature, well-run program. It was, in retrospect, almost completely meaningless.
What that number did not tell us was whether the two percent of unpatched systems included an internet-facing server actively being targeted. It did not tell us whether any of the patched vulnerabilities were ones attackers were actually exploiting in the wild, or whether the ones we spent most of our time on were the ones that mattered most from a risk standpoint. The number measured activity. It did not measure security.
That gap between activity and security is the central problem with most of the metrics programs I have encountered in this industry. We have gotten very good at counting things: patches applied, alerts triaged, training modules completed, scans run, tickets closed. We report these numbers upward, we put them on dashboards, and we use them to demonstrate that the security team is busy and working hard. All of that may be true. But busy and working hard is not the same as secure, and a metrics program that cannot tell the difference between the two is not actually measuring anything worth measuring.
The Comfort of Counting Things
Here is the honest reason most security programs default to activity metrics: they are easy to produce and easy to defend. If someone asks why you are tracking patch compliance rate, there is an obvious and intuitive answer. Patching vulnerabilities reduces attack surface. More patching is better than less patching. The number goes up, things are presumably getting safer. It has a clean logic to it that makes it very easy to present in a quarterly review without anyone asking uncomfortable questions.
Outcome metrics are harder. Asking “did our patching activity actually reduce the likelihood of a successful attack” requires you to have a much clearer picture of which vulnerabilities are being actively exploited, which of your assets are most attractive to attackers, and what your actual exposure looks like beyond just the count of unpatched systems. That is more work, more data, and more analytical complexity. It also tends to surface uncomfortable answers. And that, more than anything else, is why organizations gravitate toward the comfortable version.
As Jason Fruge, CISO in Residence at XM Cyber, wrote in a piece for The Hacker News, vanity metrics are numbers that look impressive in reports but lack real-world impact. They offer reassurance, but not insight. Meanwhile threats continue to grow more sophisticated, and attackers exploit the blind spots organizations are not measuring. [1] That framing cuts to the core of it. Vanity metrics are not harmless. They actively create a false sense of visibility that discourages the harder work of actual measurement.
Three Categories of Metrics That Look Good and Mean Little
It is worth being specific, because the problem is not that any particular metric is inherently wrong. Most of the measurements that end up as vanity metrics started as reasonable ideas. The problem is how they get used, what questions they do and do not answer, and whether leadership is ever told what the number fails to capture.
Volume metrics are the most obvious offender. These count things: scans completed, alerts generated, vulnerabilities discovered, incidents closed. They create a sense of productivity and operational tempo. They are useful for capacity planning and workload management. But they say almost nothing about whether any of that activity changed the organization’s risk profile in a meaningful way. Running more scans does not reduce risk. Acting on what those scans find in a prioritized and intelligent way is what reduces risk, and that part rarely shows up in the metric.
Time-based metrics are the second category. Mean Time to Detect and Mean Time to Respond are probably the two most universally cited security operations metrics, and both have real value when used carefully. The problem is that they get reported without context, without acknowledgment of their limitations, and without honesty about what the denominator actually includes. If your MTTD looks good because your detection logic is only firing on high-confidence, low-volume events, you are not actually measuring how fast you detect attacks. You are measuring how fast you process the subset of attacks your current detection coverage is capable of seeing. That is a very different thing, and conflating the two produces false confidence in the operations team’s capability.
Compliance and completion metrics round out the third category. Training completion rates, policy attestation percentages, audit finding closure rates. These are compliance program metrics that have been quietly promoted into security effectiveness metrics without the promotion being earned. The US Government Accountability Office, in its 2024 report reviewing the National Cybersecurity Strategy, noted that outcome-oriented performance measures largely do not exist in the cybersecurity field, and that this represents a fundamental gap in the industry’s ability to answer the foundational question of how well it is actually doing. [3] That assessment, coming from a federal auditor, is worth sitting with. After decades of security programs, the industry has still not cracked the problem of measuring what actually matters.
The Incentive Problem Nobody Talks About
The shift from activity metrics to outcome metrics is not primarily a technical challenge. The data to build better metrics is usually available. The analytical capability to interpret it exists in most mature security organizations. The real obstacle is organizational incentives, and this is the uncomfortable part of the conversation that rarely makes it into presentations about metrics programs.
Outcome metrics reveal problems. Activity metrics generally do not. If your patch compliance rate is ninety-eight percent, the number tells a clean and positive story. If you replace it with a risk-weighted metric that accounts for exploitability, asset criticality, and active threat context, you might discover that your actual exposure is far worse than the headline number suggested. That discovery demands action, which demands budget, which demands conversations nobody particularly wants to have. So the incentive structure quietly but persistently pushes organizations back toward the comfortable numbers.
There is also a gaming problem. When metrics get reported upward, teams naturally optimize for the metric rather than for the underlying reality the metric was supposed to represent. Patch compliance goes up because teams patch whatever is easiest to patch, not necessarily whatever is most dangerous. Alert triage rates improve because analysts close low-fidelity alerts faster. Phishing simulation click rates drop because the simulations get easier. The number improves. The risk does not. This is one of the most consistent and predictable failure modes in security measurement programs, and it happens because the incentives around the metric are not aligned with the outcomes the metric was designed to track. [4]
What Meaningful Measurement Actually Requires
Let me be clear about something before going further: the argument here is not that measurement is bad or that security programs should stop tracking operational data. They should not. Volume metrics, time-based metrics, and compliance metrics all have legitimate uses in the right context. The argument is that they should be used for what they are actually good for, which is operational management and workload visibility, rather than being promoted into claims about security effectiveness that they cannot honestly support.
Meaningful security measurement starts with a different question. Instead of asking “what can we count,” the question should be “what does leadership actually need to know to make good decisions about security investment and risk?” That sounds obvious, but it is a genuinely different starting point that leads to genuinely different metrics.
Gartner has framed this through what they call the CARE framework: metrics should reflect controls that are Consistent, Adequate, Reasonable, and Effective. The point is that security measurement should be grounded in outcomes that demonstrate security program credibility and defensibility, not just activity completion. [5] Translating that into practice means building metrics that answer questions like: Are we detecting adversary techniques across the attack lifecycle? Are our most critical assets protected in proportion to their value and exposure? When we do detect an incident, are we containing it before meaningful damage occurs? These questions are harder to answer than “how many patches did we apply,” but they are the questions that actually tell you something about security.
The security awareness space is a useful illustration of what this shift looks like in practice. Most organizations still measure training completion rates and phishing simulation click rates as their primary human risk metrics. Both of these measure whether a thing happened, not whether it changed anything. Completion rate measures whether employees opened a training module, not whether their behavior improved. Click rate on simulations is volatile, easy to manipulate by changing simulation difficulty, and often just measures whether the simulation was obvious enough to avoid rather than whether the underlying susceptibility changed. The meaningful version of human risk measurement tracks actual reporting rates of real suspicious activity, reduction in credential compromise incidents over time, and whether repeat offenders are improving or not. Those numbers tell you something. Completion percentages do not. [6]
The Honest Conversation Most Programs Avoid
There is a reason security metrics conversations rarely happen honestly inside organizations, and it is worth naming directly. If you go into a board meeting and say “our security program cannot currently answer the question of whether we would detect a sophisticated attacker operating in our environment,” that is an uncomfortable statement to make. It is also, for most organizations, a true one. Most security programs have significant detection coverage gaps, significant visibility gaps in cloud and hybrid environments, and limited ability to test their own effectiveness against realistic adversary behavior.
Acknowledging that honestly is the first step toward actually improving it. But it requires a level of organizational candor that the current incentive structure around metrics does not particularly reward. The teams that present reassuring dashboards full of green metrics get fewer hard questions in their quarterly reviews than the teams that present honest assessments with gaps clearly identified. That dynamic has to change before measurement programs can become genuinely useful rather than genuinely comfortable.
The most important thing I can say about this is that the problem is not the metrics themselves. Patches matter. Triage speed matters. Training happens. None of that is wrong. What is wrong is the claim that those numbers, reported in isolation without honest acknowledgment of their limitations, constitute evidence that a security program is working. They constitute evidence that the security team is busy. Those are not the same thing, and organizations that cannot tell the difference between them are flying blind in a way that their dashboards will never reveal.
Throughout this series, I am going to get into the specific mechanics of what better measurement looks like across different domains: detection coverage, vulnerability management, human risk, compliance, and program maturity. But all of that work builds on the foundation of this first post, which is simply this: you have to be honest about what your current metrics are actually measuring before you can build something more useful in their place. That honesty is harder than it sounds, and it is the work that most organizations have not done yet.
The next post in this series takes on Mean Time to Detect and Mean Time to Respond specifically, looking at what they genuinely measure, where they break down, and what a more honest version of operational security measurement looks like.
