Measuring What Matters vs. What's Measurable: The Metric Substitution Problem

The substitution

There’s a well-documented cognitive bias called attribute substitution: when faced with a hard question, people unconsciously answer an easier one instead. Organisations do the same thing with measurement.

The hard question: “Is our strategy working?” The easier question: “Are our metrics green?”

These aren’t the same question. But in every organisation we’ve studied, they’re treated as though they are. The result is a measurement system that provides false confidence — green dashboards and improving KPIs that mask strategic disconnection, declining competitive position, or eroding customer value.

The pattern is so consistent that we’ve come to treat it as a structural feature of how organisations operate, not a failure of any individual team or leader.

How substitution works

Step 1: Define what matters

The organisation articulates strategic outcomes. “Increase customer lifetime value.” “Accelerate time to market.” “Build a culture of innovation.” These are the things that actually matter — the outcomes the strategy is designed to produce.

Step 2: Proxy selection

These outcomes are hard to measure directly. Customer lifetime value requires years of observation. Time to market depends on how you define “market ready.” Culture is inherently qualitative. So the organisation selects proxy metrics: NPS for customer value, sprint velocity for time to market, number of ideas submitted for innovation culture.

Step 3: Proxy optimisation

Once the proxy is selected, the organisation optimises for it. Teams are measured on NPS scores, velocity numbers, and idea counts. Incentives, recognition, and performance reviews align to these proxies.

Step 4: Proxy divergence

Over time, the proxy and the underlying outcome diverge. NPS scores improve because the survey is sent at moments of satisfaction, not moments of truth. Velocity increases because tickets are sized smaller, not because delivery is faster. Idea counts rise because submissions are incentivised, not because innovation is occurring.

The proxy metric improves. The outcome it was meant to represent doesn’t. But the organisation has lost the ability to see the divergence because it’s only looking at the proxy.

The moment an organisation starts optimising a proxy metric, the proxy stops being a reliable indicator of the thing it was meant to measure. This is Goodhart’s Law, and it operates in every enterprise we’ve studied.

Three examples from practice

Customer satisfaction. A telecommunications company measured customer satisfaction through post-interaction surveys. Scores improved for six consecutive quarters. Customer churn increased over the same period. The surveys were sent after routine transactions (payments, simple inquiries) where satisfaction was naturally high. The moments that drove churn — service outages, billing disputes, contract friction — weren’t surveyed because those interactions didn’t have a clean endpoint for survey delivery.

Innovation. A financial services firm tracked innovation through its internal innovation lab metrics: ideas submitted, prototypes built, demos delivered. All metrics improved year over year. Over the same period, zero innovations from the lab reached production. The lab measured activity, not outcomes. The metrics rewarded the appearance of innovation, which is structurally different from the delivery of it.

Employee engagement. A consulting firm measured engagement through an annual survey. Scores were strong. Voluntary attrition was rising. The survey measured sentiment about the work environment. What was driving attrition — compensation competitiveness, career progression, and workload — wasn’t captured by the engagement instrument because those questions were considered “too sensitive” to include.

Why this persists

Metric substitution persists because it serves everyone’s interests in the short term:

Leaders get measurable evidence of progress without confronting difficult truths about strategic effectiveness. Teams get achievable targets they can influence directly. The measurement function gets clean, defensible data. Nobody has an incentive to ask whether the metric still represents the outcome — because the answer might be uncomfortable.

Breaking the pattern

The fix isn’t better metrics — it’s a different relationship between metrics and outcomes:

Maintain explicit proxy maps. For every metric, document what outcome it’s a proxy for, what assumptions link the proxy to the outcome, and under what conditions the proxy would diverge from the outcome. Review these assumptions regularly.

Measure the outcome directly, even imperfectly. Imprecise measurement of the right thing is more valuable than precise measurement of the wrong thing. If customer lifetime value is what matters, measure it — even if the measurement is lagged, estimated, or uncertain. Having it alongside the proxy prevents substitution.

Look for contradictions. When a proxy metric improves but adjacent signals suggest the outcome isn’t improving, treat the contradiction as data. NPS is up but churn is up? That’s a signal that the proxy has diverged. Don’t explain it away — investigate it.

The organisations that measure effectively aren’t the ones with the most metrics. They’re the ones that maintain the clearest line of sight between what they measure and what they’re trying to achieve.