Today's concept: why evaluating AI is hard, there's no single right answer.
Two experienced reviewers read the same AI-generated answer. One calls it great, the other calls it mediocre. Neither is wrong.
That never happens with code. Grading code is like grading a math test: one correct answer, so you write a check and run it (assert result == 5). Pass or fail.
Grading an LLM's output is like grading an essay: a summary, a chat reply, a code explanation. Many valid "correct" versions exist, and which one wins depends on tone, context, and who's reading it. You can't write a simple equality check for that. Remember it as: you're grading an essay, not a multiple-choice test.
That's the real day job behind a lot of AI engineering. "Does my agent work?" isn't pass/fail, it's a judgment call repeated thousands of times, so the field builds a test suite around the judgment itself:
1) LLM-as-a-judge: have a strong model grade a weaker model's output against a rubric. G-Eval (EMNLP 2023) formalized this: GPT-4 reasons step by step through a rubric, then scores. It matched human judgment far better than word-overlap metrics like BLEU or ROUGE.
2) The judge has its own biases. The MT-Bench / Chatbot Arena paper (Zheng et al., NeurIPS 2023) found signs GPT-4 rated its own answers roughly 10% higher than a human judge would, a pattern called "self-enhancement bias." The same paper found judges also favor whichever answer comes first (position bias) and reward longer answers regardless of quality (verbosity bias).
3) Different jobs need different harnesses. A RAG (retrieval-augmented generation) system gets graded on whether its answer is backed by what it retrieved (RAGAS, an open framework, calls this "faithfulness"). An agent gets graded on whether it used the right tools in the right order. The highest-stakes cases still go to human review. Even inference now borrows this idea: reasoning models sample several candidate answers and use a verifier, or "process reward model," to pick the best one live, the "test-time compute" trend you've likely seen mentioned.
Why it matters even if you don't build AI: every product review, code assistant, and support bot you use is only as good as the invisible judge that decided what "good" meant during training and testing.
Quick check before you scroll: Why can't you just write a normal unit test (assert output == expected) to check if an LLM's response is "correct"?
Full breakdown + the answer: frankduah.me/learnings/2026-07-27-why-evaluating-ai-is-hard-no-single-right-answer
New here? I post a bite-size AI / ML concept like this every day, follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#EvaluatingAI #AI #LLM #AIAgents #MachineLearning
The answer
Because most LLM outputs (summaries, chat replies, explanations) have many valid correct versions, not one fixed answer, so there's no single ground truth to assert equality against. You need judgment (a human or another LLM acting as judge) instead of an exact match.