METR, a nonprofit that tests AI models, gave OpenAI's o3 coding tasks marked by

Infographic: what AI alignment means. A model does what its score rewards, not what you meant. Four steps, top to bottom: you want an outcome, you score a stand-in, the model chases the score, it finds the holes. Examples: CoastRunners and METR's o3 test. Rule: whatever you measure gets optimized. — Loading image...

METR, a nonprofit that tests AI models, gave OpenAI's o3 coding tasks marked by an automatic grader. In some runs it skipped the work and edited the grader so it always reported a perfect score.

It wasn't confused, either. On another task, where it copied the grader's own answer, METR asked o3 whether that matched what the user wanted, and it said no.

That gap has a name. AI alignment is the work of getting a model to do what its builders actually meant, not just what its score rewards.

The gap exists because a model learns by chasing a number. That number is either a loss (how wrong its output was) or a reward (a score from a person or a program saying how good the output was). It's only a stand-in for what you want, and when the stand-in has holes, the model finds them. That's called reward hacking.

The classic case comes from OpenAI in 2016. An AI learned to play CoastRunners, a boat racing game that gave points for hitting targets along the course. The researchers assumed points would track finishing the race. Instead the boat found a lagoon where targets kept reappearing and circled it over and over, catching fire and crashing into other boats. It never had to finish the course, and it still outscored human players.

So here's the rule worth keeping: whatever you measure is what the model will optimize. If your score can be gamed, assume it will be.

This gets practical fast once you build an AI agent (a model that takes actions, like running code or editing files). Your prompt, your success checks and your tests become its reward. An agent that "passes the tests" by editing the tests is CoastRunners in miniature. A clearer prompt won't fix that. Keep the grading code and the tests somewhere the agent can't touch.

Quick check before you scroll: Why did the CoastRunners boat score high without ever finishing the race?

Full breakdown + the answer: frankduah.me/learnings/2026-10-22-ai-alignment-in-plain-english

New here? I post a bite-size AI / ML concept like this every day. Follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH

#AIAlignment #AISafety #AI #LLM #AIAgents #MachineLearning

The answer

The game paid points for hitting targets, not for finishing, so the boat found a loop that farmed targets forever. The score was a flawed stand-in for the real goal.

- / Learnings

- Learnings