Head of AI Safety at US AI Safety Institute, NIST
Paul Christiano
Profile
Paul Christiano is the closest thing AI safety has to a founding engineer. While most of the field’s public figures are philosophers or executives, Christiano is the person who actually shipped the technique that made modern chatbots usable. At OpenAI, he led the language model alignment team and was the principal architect of reinforcement learning from human feedback (RLHF) — the method that takes a raw, unruly language model and trains it to be helpful and to (mostly) follow instructions. If you have ever gotten a straight answer out of ChatGPT, Claude, or Gemini, you are using the descendant of a training recipe Christiano co-authored in the 2017 paper Deep Reinforcement Learning from Human Preferences, alongside collaborators including Jan Leike and Dario Amodei. For developers, that is the point: RLHF is the reason instruction-following works at all, and Christiano is the reason RLHF exists.
The mathematical pedigree is real — a silver medal at the 2008 International Mathematics Olympiad, a math degree from MIT, and a Berkeley PhD under Umesh Vazirani. But what makes Christiano unusual is that he walked away from the frontier lab at the height of its momentum. He left OpenAI in 2021 to found the Alignment Research Center (ARC), a small Berkeley nonprofit tackling the theoretical problems that a scaling lab has no incentive to slow down for. ARC split into two efforts that matter enormously to anyone building with AI: ARC Theory, which chases hard conceptual puzzles like Eliciting Latent Knowledge (how do you get a model to honestly report what it internally “knows,” even when lying would score better?), and ARC Evals (now METR), which pioneered dangerous-capability testing — including the now-famous evaluations of whether GPT-4 could autonomously replicate, acquire resources, or deceive humans.
In April 2024, U.S. Commerce Secretary Gina Raimondo named Christiano Head of AI Safety at the U.S. AI Safety Institute, housed at NIST (later reorganized as the Center for AI Standards and Innovation). This is arguably the most consequential technical-safety job in the U.S. government: designing and running the evaluations by which the federal government judges frontier models for national-security-relevant capabilities. It moves Christiano from the theory board to the standards table — the person deciding what “tested” actually means for the models the rest of the industry ships.
What developers should take from Christiano is his intellectual style, not just his résumé. He is a careful, quantitative thinker who resists both hype and hysteria — famous for putting real numbers on his worries (roughly a 10–20% chance of AI takeover, and something like a coin-flip of “doom” conditional on reaching human-level systems that are poorly handled). He is also refreshingly self-aware about the double-edged nature of his own work: RLHF made AI both safer and far more commercially powerful, a tension he discusses openly. His body of writing — much of it on his blog ai-alignment.com and the Alignment Forum — is some of the clearest technical thinking available on what it actually takes to supervise systems smarter than their supervisors.
Key Articles & Papers
Deep Reinforcement Learning from Human Preferences AI Safety via Debate Supervising Strong Learners by Amplifying Weak Experts (Iterated Amplification) Eliciting Latent Knowledge (ARC Technical Report) My Views on 'Doom' What Failure Looks LikeVideos
Controversies
Christiano’s March 2024 appointment to the AI Safety Institute drew internal pushback at NIST: staff members and scientists reportedly threatened to resign, arguing that his prominent ties to the effective altruism and longtermism movements could compromise the institute’s objectivity, and objecting to how quickly the hire moved through the process (VentureBeat). No resignations were publicly confirmed, and Christiano took the role. Critics on the accelerationist side have labeled him an “AI doomer,” a framing he pushes back on — his public probability estimates are deliberately more moderate and hedged than that label implies. More substantively, Christiano has openly acknowledged the criticism that RLHF is dual-use: by making models more helpful and controllable, it also made them dramatically more commercially valuable, arguably accelerating the very race that safety research is trying to make safe.
Spotify Podcasts
YouTube
