https://video.ut0pia.org/videos/watch/9705a607-2dd7-4df9-bf1d-8af561fdad0f David Brumley has spent two decades turning people into hackers, from founding picoCTF to recruiting pwn2own winners at Carnegie Mellon, and his argument is that you teach a model to hack the same way: a ladder of tasks that climbs from triggering a crash to reading and writing arbitrary memory to a full working exploit. The catch is measurement. Hacking has no single answer, so the usual benchmark setup breaks down when a target has multiple vulnerabilities and a language model can always claim it found one, and grading oracles that just ask the model whether it succeeded are hopeless. So Brumley's team builds real reinforcement learning environments instead: reproducible, sandboxed, and scored by deterministic graders that check whether an exploit actually triggers the specific bug, borrowing precision and recall from his DARPA Cyber Challenge work where he designed the scoring. He shows it on V8, the JavaScript engine in Chrome, running against 41 real vulnerabilities where the strongest models reached about 95% and, in the hard cases, produced genuine out of sandbox exploits including a real zero day. The point that lands is a warning against benchmaxxing security: build environments grounded in real bugs and honest graders, because that is what separates a model that looks like it can hack from one that actually can. Speaker info: https://www.linkedin.com/in/thedavidbrumley, Timestamps: 0:00 - Two decades of teaching hacking 1:54 - From CTF scoreboards to CMU 3:34 - A ladder of exploitation tasks 6:44 - Why measuring hacking is hard 7:46 - Flawed grading oracles 10:30 - When a target has many bugs 13:22 - Deterministic graders and AIXCC scoring 14:49 - Precision and recall for vulnerabilities 17:35 - Attacking V8 in Chrome 21:10 - 41 vulnerabilities and a real zero day 25:24 - Don't benchmaxx security Sat, 01 Aug 2026 18:00:07 GMT https://validator.w3.org/feed/docs/rss2.html PeerTube - https://video.ut0pia.org https://video.ut0pia.org/lazy-static/avatars/0287a09a-aae7-4840-9843-b416426e7046.webp https://video.ut0pia.org/videos/watch/9705a607-2dd7-4df9-bf1d-8af561fdad0f All rights reserved, unless otherwise specified in the terms specified at https://video.ut0pia.org/about and potential licenses granted by each content's rightholder.