To rank the world's biggest supercomputers, we're gonna need a bigger bench
The Top500's biggest system runs 2.2 exaFLOPS, but AI clusters like Colossus and Stargate pack far more GPUs and still can't be ranked against it.
Status
- Announced
- Demonstrated
- Available
- Independently confirmed
Sources
For three decades, the Top500 list has ranked the world's biggest publicly known supercomputers by a single yardstick: how fast they run High-Performance Linpack, a dense linear algebra benchmark in double-precision math. That yardstick now measures a shrinking slice of the planet's real compute.
The current Top500 leader, China's Lineshine, hits 2.2 exaFLOPS of double-precision performance on HPL. Set that next to SpaceXAI's Colossus, Meta's Promethius, or OpenAI's Stargate, each built from hundreds of thousands of GPUs, and Lineshine looks like a rounding error in raw hardware terms.
The gap exists because the hyperscalers building those clusters don't submit to the list at all. "These [companies] are not interested in submitting to the Top500," ETH Zurich's Torsten Hoefler said during his Supercomputing conference address this week.
Top500 co-founder Erich Strohmaier frames the mismatch as a split between capability and capacity. For most of the list's history the two tracked together: a system built to run one enormous calculation across its full width was also, by construction, among the largest machines on Earth. AI broke that correlation. "What happened in the last few years with AI is capacity really pulled ahead of capability in terms of market size," Strohmaier said. "We were and we still are very focused on capability."
Capability, in Top500 terms, means running a single piece of code across the entire system at once. "Our prime rule for benchmarking is, you have to benchmark your whole system with one code, you cannot just add two parts up to make up a number," Strohmaier said.
AI training clusters fail that test by design. "All those systems are capacity systems," Strohmaier said, built to run many small or medium-sized jobs in parallel: training runs, fine-tuning passes, reinforcement learning loops, inference requests, often all at once rather than one giant unified computation. Asking SpaceXAI to run HPL across the whole of Colossus wouldn't reflect how the machine actually operates, Strohmaier said.
That leaves the Top500's organizers with a measurement gap and no easy fix. Hoefler is wary of touching the underlying benchmark. "If we do anything to the benchmark itself, we're going to lose the historic trends, which is extremely valuable," he said, pointing to HPL's three-decade run as the thing worth protecting.
One option already has a foothold: loosen the precision requirement. HPL-MxP, a mixed-precision variant introduced in recent years, lets systems get credit for lower-accuracy math, which matters because modern accelerators increasingly trade away FP64 throughput for FP8 and FP4 performance tuned for AI workloads.
The other option is a dedicated AI benchmark built from scratch for these clusters, one that could apply to Colossus and Stargate right away. Model architectures move too fast for that to hold up over time. "The AI workloads change every two weeks, so if you pick something today, it's going to be irrelevant in at most a year, and these benchmarks need to live for multiple decades," Hoefler said.
No replacement benchmark exists yet. Until one does, the world's largest AI clusters will keep running outside the ranking built to track the world's largest machines.
Keep reading
- Nebius's $40 Billion Number Is Backlog, Not Spending, and the Gap Is the Story
- Yandex Lost Nearly 3 Percent of Cloud Data to Drone Strikes, Now Makes Backups the Default
- IBM to offer access to a dual-processor quantum system in 2027
- Google's orbital AI satellite writes a haiku, and a very expensive one