Proxy Metrics Under Optimisation: Why a Correlation of 0.6 Is Not a Substitute for the Goal

The metric the business cares about arrives too late and too noisily to optimise: renewal, satisfaction, long-term retention. So the team picks a proxy that arrives immediately, checks that it correlates 0.63 with the goal across the existing catalogue, and starts ranking by it. Six months later the proxy is up by 3.8 units and the goal is up by 1.9.

Nothing was faked and no bug was introduced. The correlation of 0.63 was real, and it was computed on a population nobody was optimising. Once the optimiser starts choosing items by the proxy, it stops sampling that population and starts sampling the top of it, and the top of a proxy is enriched in whatever is easiest to raise, which is not the same as what the goal rewards.

A Proxy Is a Sum

Write the proxy as the thing you want plus the part that moves it without moving the goal:

Here $q$ is quality, which the goal rewards; $m$ is the manipulable component, clickbait in a headline, padding in an answer, urgency in a subject line, anything that raises the proxy on its own; and $c$ says whether that component is merely useless to the goal ($c = 0$) or actively costly ($c > 0$). The observed correlation between $P$ and $Y$ depends on how much variance $m$ contributes, and it can look reassuring while $m$ is doing most of the work at the top of the distribution.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import numpy as np

rng = np.random.default_rng(0)

def pool(n, var_m=1.0, cost=0.0, r=rng):
    """Proxy P = q + m; goal Y = q - cost * m. The manipulable part m raises the proxy and
    is worthless (cost 0) or harmful (cost > 0) to the goal."""
    q = r.normal(0, 1, n)
    m = r.normal(0, np.sqrt(var_m), n)
    y = q - cost * m + r.normal(0, 0.5, n)
    p = q + m
    return q, m, y, p

for var_m, cost in ((0.25, 0.0), (1.0, 0.0), (4.0, 0.0), (1.0, 0.5), (1.0, 1.0)):
    q, m, y, p = pool(200000, var_m, cost)
    print(f"var(m) {var_m:.2f}, cost {cost:.1f}: corr(proxy, goal) = {np.corrcoef(p, y)[0, 1]:+.2f}")

The last row is the one to hold on to. A proxy can be uncorrelated with the goal in the population and still be the thing everyone optimises, because the correlation that gets checked is usually computed on a sample where $q$ dominates, or long before $m$ had a reason to grow.

What Optimising It Delivers

Selecting the best items by the proxy raises the proxy by a known amount. How much of that is real depends on how the proxy's variance splits.

1
2
3
4
5
6
7
8
9
for var_m in (0.25, 1.0, 4.0):
    for frac in (0.5, 0.1, 0.01):
        gp, gy = [], []
        for _ in range(200):
            q, m, y, p = pool(20000, var_m)
            sel = np.argsort(p)[-int(len(p) * frac):]
            gp.append(p[sel].mean() - p.mean()); gy.append(y[sel].mean() - y.mean())
        print(f"top {frac:.0%}, var(m) {var_m}: proxy +{np.mean(gp):.2f}, goal +{np.mean(gy):.2f}, "
              f"real share {np.mean(gy)/np.mean(gp):.0%}")

The real share is constant within each block and equals $\operatorname{var}(q)/[\operatorname{var}(q) + \operatorname{var}(m)]$: 80 percent at $\operatorname{var}(m) = 0.25$, 50 percent at 1, 20 percent at 4. That is the quantity that governs a proxy under optimisation, and it is not the correlation. At $\operatorname{var}(m) = 1$ the correlation is a respectable 0.63 while only half the reported gain is real; at $\operatorname{var}(m) = 4$ the correlation is 0.40 and four fifths of every reported gain is an artefact of selection.

For comparison, selecting on the goal metric itself gains 0.89, 1.96 and 2.98 at the three selection levels. The proxy at $\operatorname{var}(m) = 1$ delivers 0.56, 1.24 and 1.88: about 63 percent of what direct optimisation would have achieved, which is the correlation making its one honest appearance.

The Correlation Does Not Survive Selection

If the proxy-goal correlation is being monitored to confirm that the proxy is still a good target, it will look worse and worse, and the decline is a property of selection rather than evidence of a change in the world.

1
2
3
4
5
6
7
for frac in (1.0, 0.5, 0.1, 0.01):
    cs = []
    for _ in range(200):
        q, m, y, p = pool(20000, 1.0)
        sel = np.argsort(p)[-int(len(p) * frac):] if frac < 1 else np.arange(len(p))
        cs.append(np.corrcoef(p[sel], y[sel])[0, 1])
    print(f"top {frac:.0%}: corr among the selected {np.mean(cs):+.2f}")

Restricting to high-proxy items removes most of the variation in the proxy and leaves the two components trading off against each other within the set, which is the same mechanism that makes a selected sample look uncorrelated in any restricted range. A team that recomputes the correlation on shipped items and concludes the proxy has "degraded" is measuring the selection it performed.

More Search Is More Pressure

The share that is real is fixed by the variance split, but the absolute damage grows with how hard the optimiser searches. Holding the output fixed at ten items and enlarging the pool they are chosen from is the cleanest way to see it.

1
2
3
4
5
6
7
8
for n in (100, 1000, 10000, 100000):
    gp, g0, g1 = [], [], []
    for _ in range(200):
        q, m, y, p = pool(n, 1.0); sel = np.argsort(p)[-10:]
        gp.append(p[sel].mean() - p.mean()); g0.append(y[sel].mean() - y.mean())
        q, m, y, p = pool(n, 1.0, cost=1.0); sel = np.argsort(p)[-10:]
        g1.append(y[sel].mean() - y.mean())
    print(f"{n:>7,} candidates: proxy +{np.mean(gp):.2f}, goal +{np.mean(g0):.2f} (cost 0), {np.mean(g1):+.2f} (cost 1)")

A thousand-fold larger search more than doubles the reported gain and delivers a little over double the real one when the manipulable part is harmless. When it carries a cost equal to its benefit, the entire gain from a hundred-thousand-candidate search is zero: the optimiser has become very good at finding items whose proxy advantage is exactly the thing the goal penalises. This is the mechanism behind the observation that a metric stops working when it becomes a target, and it sharpens it: what breaks the metric is not the target itself but the amount of search applied to it.

When Optimisation Goes Backwards

The cost parameter decides whether hard optimisation is merely inefficient or actively harmful.

1
2
3
4
5
6
7
8
9
10
for frac in (1.0, 0.5, 0.25, 0.1, 0.01):
    row = []
    for cost in (0.0, 0.5, 1.0, 2.0):
        gy = []
        for _ in range(200):
            q, m, y, p = pool(20000, 1.0, cost)
            sel = np.argsort(p)[-int(len(p) * frac):] if frac < 1 else np.arange(len(p))
            gy.append(y[sel].mean() - y.mean())
        row.append(np.mean(gy))
    print(f"top {frac:.0%}: " + "  ".join(f"{v:+.2f}" for v in row))

Goal gain by selection intensity and by how costly the manipulable component is.

With the components of equal variance, a cost of 1 is the break-even point: optimising the proxy as hard as you like moves the goal by exactly nothing, while the proxy reports a gain of 3.77. Above it, every increment of optimisation makes the product worse while the dashboard improves. The team is not being deceived by anyone; it is reading a number that was never measuring the thing it names.

A Small Holdout Settles It

None of this is detectable from the proxy, and all of it is detectable from a sample measured on the goal. Because the effect of hard optimisation on the goal is large when it is harmful, the holdout can be tiny.

1
2
3
4
5
6
7
8
9
10
11
for n_hold in (50, 200, 800, 3200):
    detect = 0
    for _ in range(400):
        q, m, y, p = pool(20000, 1.0, cost=2.0)
        sel = np.argsort(p)[-200:]
        base = rng.choice(len(y), n_hold, replace=False)
        take = rng.choice(sel, min(n_hold, len(sel)), replace=False)
        diff = y[take].mean() - y[base].mean()
        se = np.sqrt(y[take].var(ddof=1) / len(take) + y[base].var(ddof=1) / len(base))
        detect += (diff / se) < -1.96
    print(f"{n_hold} per group: goal significantly worse in {detect / 400:.0%} of runs")

Fifty items measured on the goal metric, against fifty random ones, catch a harmful optimisation virtually every time. The cost of that check is a rounding error against the cost of running the optimisation, and the reason it is so often skipped is not expense but that the proxy was chosen precisely because measuring the goal was inconvenient.

Making the Proxy Harder to Exploit

Two structural defences follow directly from the decomposition. The first is to combine proxies whose manipulable components differ, so that raising one does not raise the others.

1
2
3
4
5
6
7
8
9
for k in (1, 2, 4):
    gp, gy = [], []
    for _ in range(200):
        q = rng.normal(0, 1, 20000); y = q + rng.normal(0, 0.5, 20000)
        ms = [rng.normal(0, 1, 20000) for _ in range(k)]
        ps = np.mean([q + m for m in ms], axis=0)
        sel = np.argsort(ps)[-200:]
        gp.append(ps[sel].mean() - ps.mean()); gy.append(y[sel].mean() - y.mean())
    print(f"{k} proxies averaged: proxy +{np.mean(gp):.2f}, goal +{np.mean(gy):.2f}")

Averaging four proxies with independent manipulable parts raises the realised goal gain from 1.88 to 2.39, most of the way to the 2.98 that direct optimisation of the goal would have produced, because the average of independent manipulable components has a quarter of the variance while the shared quality component survives intact. The second defence is to cap or penalise the proxy rather than maximise it: a target that says "at least this much" stops the search before it reaches the region where only $m$ is left to give.

What to Do

1. Write the proxy as quality plus the manipulable part, and estimate the share of its variance that is quality. That share, not the correlation, is the fraction of any reported gain you should expect to be real.
2. Measure the goal on a holdout whenever optimisation pressure changes, and size it from the harm you would want to catch; fifty items per group detects a serious regression here.
3. Expect the proxy-goal correlation to fall among selected items and do not read the fall as evidence about the world.
4. Treat search effort as a dial on the risk. The same proxy is safe under light selection and dangerous under heavy selection; a larger candidate pool raises pressure without any change in the metric.
5. Combine proxies with different failure modes, or cap rather than maximise, when the goal cannot be measured often enough to govern the optimiser directly.
6. Re-derive the proxy when the system changes. A proxy validated before optimisation says nothing about the regime that optimisation creates.

References

• Goodhart, C. A. E. (1984). Problems of monetary management: the UK experience. In Monetary Theory and Practice. Palgrave Macmillan.
• Manheim, D., & Garrabrant, S. (2018). Categorizing variants of Goodhart's law. arXiv preprint, arXiv:1803.04585.
• Prentice, R. L. (1989). Surrogate endpoints in clinical trials: definition and operational criteria. Statistics in Medicine, 8(4), 431-440.
• Fleming, T. R., & DeMets, D. L. (1996). Surrogate end points in clinical trials: are we being misled? Annals of Internal Medicine, 125(7), 605-613.
• Dmitriev, P., Gupta, S., Kim, D. W., & Vaz, G. (2017). A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1427-1436.
• Deng, A., & Shi, X. (2016). Data-driven metric development for online controlled experiments: seven lessons learned. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 77-86.
• Gao, L., Schulman, J., & Hilton, J. (2023). Scaling laws for reward model overoptimization. Proceedings of the 40th International Conference on Machine Learning, 10835-10866.
• Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.