Anthropic researcher, weak-to-strong generalization and AI alignment
Jan Leike
Profile
Jan Leike is one of the few researchers whose career doubles as a map of where AI alignment has actually been done. A theoretical computer scientist by training — he earned a PhD in reinforcement learning theory from the Australian National University and did a stint at the University of Oxford — Leike turned to the practical, empirical side of deep learning at exactly the moment it started to matter. At DeepMind he helped prototype reinforcement learning from human feedback (RLHF), the technique that would later make ChatGPT usable. If you have ever wondered why a base language model and a chat assistant behave so differently, the lineage runs straight through Leike’s early work.
At OpenAI he was head of alignment and worked on the alignment of InstructGPT, ChatGPT, and GPT-4 — then in mid-2023 he and Ilya Sutskever were named co-leads of the Superalignment team, a four-year, 20%-of-compute bet on controlling systems smarter than their supervisors. In May 2024 that bet collapsed publicly. Leike resigned days before OpenAI dissolved the team, writing in a widely-read thread that “safety culture and processes have taken a backseat to shiny products” and that he had “gradually lost trust” in leadership. For developers watching from outside, it was the clearest signal yet that the safety-versus-velocity tension inside frontier labs was real, not rhetorical.
Within two weeks he had landed at Anthropic, where he now leads the Alignment Science team. The through-line of his research is a bet that most people still find counterintuitive: instead of hoping humans can supervise superhuman systems directly, build AI that helps you do the alignment research itself. His team’s agenda — scalable oversight, weak-to-strong generalization, robustness to jailbreaks, and an “automated alignment researcher” — is arguably the most technically concrete safety program running at any lab, and it increasingly sets the terms of debate for academic groups too.
What makes Leike worth understanding if you are building with AI today is that his work is not abstract doom philosophy. It is about a problem you already have in miniature: how do you get a model to do the right thing on tasks you cannot easily check yourself? RLHF, critique models, and weak-to-strong methods are the tools he has helped invent to answer that, and they are the same tools shaping the assistants you use every day.
Key Articles & Papers
Deep Reinforcement Learning from Human Preferences Scalable Agent Alignment via Reward Modeling Training Language Models to Follow Instructions with Human Feedback (InstructGPT) Weak-to-Strong Generalization LLM Critics Help Catch LLM Bugs A Minimal Viable Product for Alignment Why I'm Optimistic About Our Alignment Approach What Could a Solution to the Alignment Problem Look Like?Videos
Controversies
Leike’s May 2024 departure from OpenAI is the defining public episode of his career, and it is best read as a principled disagreement rather than a scandal. Resigning alongside Sutskever and other safety staff, he argued in a public thread on X that OpenAI’s safety work was being starved of compute and attention relative to product launches. OpenAI leadership publicly acknowledged there was “a lot more to do” and committed to it; critics of Leike countered that going public amplified distrust in ways that helped no one. Either way, his exit — and his immediate move to a direct competitor — became a flashpoint in the broader argument over whether frontier labs can police themselves, and it is worth understanding on its own terms rather than as a takedown of any single company.
Spotify Podcasts
YouTube
