A tiny live demo of the SIA paper's Goodhart? warning: a self-improving loop can raise its verifier score while getting worse on held-out reality if the verifier is bad, too easy, or overfit.
Topicsbad verifiereasy verifieroverfit verifierrobust guardrailAnswer Engine Optimization application
SIA's biggest practical bottleneck is verifier design. Harness updates and weight updates are powerful, but both optimize the scoring function you give them.
If the score is a proxy, the optimizer learns the proxy. If the score leaks the training examples, it learns the leak.
The demo separates the visible-to-optimizer verifier score from the hidden reality score used to catch score gaming. The goal is not one perfect formula; it is a verifier/evaluation loop that keeps finding where the current score diverges from reality.
The public scoring rule that produces the visible score the optimizer is allowed to optimize. If this rule is shallow, the winner can look good while being wrong.
An independent held-out score the optimizer does not see. It measures whether the selected answer generalized or merely gamed the visible verifier score.
memorized_training_answer?
This is benchmark leakage: exact visible examples get rewarded, but paraphrases and new cases fail.
In this toy demo, Robust is hardcoded because we know the failure modes. In real systems, hidden tests, source grounding, freshness checks, adversarial prompts, and human review must keep evolving.
Both optimizers see the same candidate mutations for 12 loops. The visible-only optimizer picks the highest active verifier score. The independent-verifier-gated optimizer can still optimize that visible score, but it rejects candidates that fail a separate independent gate score. Hidden reality audit is reported after selection and is not used to choose winners.
Each row shows the optimizer's choice after that loop. Rejections are candidates that looked better to the visible verifier but failed the independent verifier gate.
Borrowing the SIA-Lever lesson: after a failure, the meta-agent should choose whether to fix the harness/verifier, train weights, or do both. The dangerous move is to train weights against a bad verifier.
This trace turns the abstract Goodhart warning into a concrete sequence: answer → verifier score → optimizer choice → hidden audit → lever decision. It updates with the active verifier lens.
Toggle between the weak proxy and robust guardrail to see why the safe move changes. Weak verifier: visible score can rise while hidden reality fails. Robust verifier: score and hidden audit are aligned enough that W can be attempted under audit.
Use these as transfer examples: the toy score gap maps to real agents that pass visible checks while failing the actual job.
A compact builder checklist for deciding whether to repair H, train W, or do H→W.
Move verifier strictness, hidden audit strength, and optimizer pressure. The toy calculator shows how a high visible score can still produce a large hidden-reality gap and blind W regret.
Train weights only after the harness is good enough that optimizing against it improves reality, not just the metric.
The demo is not saying “never train W.” It is saying: if the verifier rewards shortcuts, fix H first. Once the visible score reliably tracks held-out audits, W becomes a useful capability lever again.
Answer the mini quiz. Each question is about the operational lever, not just the score.
This public demo borrows the lever-attribution idea conceptually: decide whether to repair the harness (`H`), train weights (`W`), or repair the harness before training (`H→W`). No upstream source code or prose was copied.
Verifier scores update when you switch lenses. The hidden reality score is independent and stays fixed for each answer style.
| Verifier | Optimized winner | Verifier score | Hidden reality score | Gap |
|---|
If Answer Engine Optimization optimizes the wrong metric, it can Goodhart too: