https://video.ut0pia.org/videos/watch/8dc9bb8b-14f6-412d-9299-2a85f601ed76 Every time a model launches there is a gap between the benchmark numbers and what the thing can actually do, and Nick Heiner argues the existence of the word benchmaxxing is the tell. When labs openly brag about scores, teams stop asking whether a benchmark reflects reality, and the whole field drifts into an avalanche of numbers that measure the wrong thing. His talk is a field guide to reading a benchmark fairly, starting from the antipatterns that quietly break them. The failure modes are specific. A large share of tasks in a typical benchmark are simply broken; contamination means models have memorized test content, so a SWE-bench style score partly measures recall; and reward hacking lets a lazy policy satisfy the verifier without doing the task. The nastiest is misalignment between the prompt and the grader, like an eval that asks for no commas and an answer in Hindi at once, or a verifier whose sentence splitter cannot parse the format, so the only way to a perfect score is to game it. Heiner's prescription is to bring domain expertise, align tools with prompts, and pay for real human evaluation, holding both benchmark writers and the labs to a higher standard. Speaker info: https://x.com/nickheiner, https://www.linkedin.com/in/nick-heiner-3874055a/, https://www.nickheiner.com/, Timestamps: 0:00 - The benchmark versus reality gap 0:55 - Why the word benchmaxxing exists 2:38 - Reading a benchmark fairly 3:14 - Antipattern: broken tasks 4:41 - Antipattern: contamination 5:57 - Antipattern: reward hacking 6:23 - Misaligned prompts and verifiers 10:40 - Benchmaxxing as a two way street 13:14 - Domain expertise and getting it right 15:47 - Human eval and a higher standard Sun, 02 Aug 2026 22:54:37 GMT https://validator.w3.org/feed/docs/rss2.html PeerTube - https://video.ut0pia.org https://video.ut0pia.org/lazy-static/avatars/0287a09a-aae7-4840-9843-b416426e7046.webp https://video.ut0pia.org/videos/watch/8dc9bb8b-14f6-412d-9299-2a85f601ed76 All rights reserved, unless otherwise specified in the terms specified at https://video.ut0pia.org/about and potential licenses granted by each content's rightholder.