OpenAI Says It's Changing How It Shares Breakthroughs. Researchers Still Want to See the Proof.
OpenAI says new math, computer science and physics results came from "shepherds" vetting its claims, but researchers tell The Verge the company still hasn't opened those results to independent reproduction.
OpenAI announced on Monday that it had produced new results in mathematics, computer science, and physics. The company also described what it called an evolving approach to sharing its research. Researchers who spoke with The Verge's Robert Hart said they remain skeptical of the company's methods, even as OpenAI tries to adjust them.
The announcement follows a rocky stretch between OpenAI and the mathematics community. In early October, OpenAI released hundreds of preprints at once, and mathematicians spent weeks afterward trying to sort through the backlog and figure out which claims held up. That episode left many researchers wary of how OpenAI handles its own findings before they reach outside scrutiny.
This time, OpenAI says it worked with expert "shepherds", outside specialists brought in to help decide how to present the new math, computer science, and physics results. The shepherd model is OpenAI's attempt to avoid a repeat of October, when the sheer volume of unreviewed preprints made it hard for anyone to separate genuine advances from noise.
That distinction, between a claim and a verified result, sits at the center of why researchers remain unconvinced. A model card or a company blog post describes what a system produced. It doesn't establish that the result survives independent reproduction, peer review, or the kind of scrutiny that mathematics and physics have relied on for decades. Shepherds selected and paid by OpenAI can shape how a finding is framed, so the arrangement doesn't substitute for publication in a peer reviewed venue or replication by a lab with no stake in the outcome.
The pattern Hart describes is familiar from the October preprint dump: an announcement arrives, the underlying material follows later or incompletely, and the research community is left to reconstruct the evaluation after the fact. Researchers can say a result is impressive once they've seen the full preprint, the proof, and the code. They can't say that from a press release alone.
OpenAI frames the shepherd process as a sign it has learned something from the fall's backlash. Researchers quoted by The Verge describe their patience as worn down after months of announcements that outrun the documentation needed to check them. The company says this round reflects an evolving approach to communicating its findings. Whether that approach changes the underlying pattern depends on what comes next, specifically whether the math, computer science, and physics claims from Monday arrive with full preprints, open code, and enough detail for outside groups to attempt reproduction.
Until those materials are available and checked, the appropriate record is narrow. OpenAI says its systems produced new results in three fields. Outside researchers haven't yet verified those results independently. That gap, between what a company states its model did and what the field has confirmed, is the one Monday's announcement did not close.
Keep reading
- FairStress Finds Language Models Abandon Correct Answers Unevenly, Not in the Direction You'd Expect
- OpenAI's Own Models Hacked Websites and a Crypto Wallet, and the Company Wrote It Down
- HiRO-ACEv2 Promises Cheap 3 km Storm Simulation, But Its Extremes Are Still Unverified Against Reality
- Falcon-Emirati's Dialect Score Is Real, Self-Graded, and Still Unverified