Fireworks AI's pitch: stop betting on one model, start tracking cost per task

Fireworks AI sells managed inference: send it a prompt, and its routing layer picks which model answers, whether that's an open weight model it hosts itself or a frontier API from Anthropic or another lab. Rob Ferguson, a vice president at the company, told The Register the math behind that pitch in blunt terms. Ten thousand GPUs represent a billion dollars of capital, he said, and idle GPUs are money bleeding out.

That's the argument Fireworks makes against running your own inference stack on-prem. The company still supports it, and offers a product called Sovereign for customers with workloads sensitive enough to need dedicated or air-gapped infrastructure, often paired with partners like Microsoft. For most workloads, Ferguson said, that level of isolation is overkill, and the company would rather sell a portfolio approach through its Nexus platform and API.

"The state of the art for everyone is to have a portfolio of models," Ferguson said. "The best model is literally a portfolio, and we can prove it. You are always better routing between a few models than to do one singular model." That puts Fireworks in the same lane as Together AI, Baseten, and DeepInfra, all of which sell some version of multi-model routing rather than access to a single lab's API.

What separates a router from a plain gateway, according to Ferguson, is that Fireworks grades each incoming task for difficulty and matches it to a model's known strengths before sending it anywhere. "We build custom routers," he said. "Our auto model is continuously updating so that we actually go through and we grade the difficulty of the task and we see if there're affinities that the models have, like Kimi is better at JavaScript than Python."

Ferguson traces that engineering focus to the founders' history working on PyTorch, the open source machine learning library. Running a mixture of experts model well means routing each query back to the exact machine holding the right expert weights, he said, and getting that wrong doesn't break the output. It produces a different one, that can degrade quality in ways a customer won't necessarily catch. Handling floating point precision correctly across a fabric that spans more than one provider is, in his telling, where Fireworks earns its margin over buying raw GPU capacity from a hyperscaler and managing the plumbing yourself.

The economics argument rests less on sticker price than on a number Ferguson says most customers don't track: cost per task. Caching, he said, is the biggest lever most companies ignore. Fireworks was working with a data science company and caching about 90 percent of its tasks, he said, then compared that to Anthropic's Opus model on the same workload. "We looked at Opus and it was caching 30 percent of the time," he said. "It was doing AI rework constantly. And with AI, rework is by far your biggest cost. If it's not hitting a cache, then your costs are absolutely insane."

Ferguson's complaint is that frontier labs don't publish their caching rates, so a customer comparing token prices across providers is comparing the wrong number. "The token cost is not the real cost. It's the cost per task. And they're hiding that from you," he said. "So we always tell our customers to be tracking cost per task, because that's the real number."

Data retention is the other line item Ferguson wants customers to put a price on. Fireworks runs a zero data-retention policy, he said, while other providers keep what gets sent through their APIs and may use it for training. "I think people grossly underestimate the value of their data," he said. "Nobody really knew at the beginning of this current era that these models are memorizers. But now there's plenty of evidence that these models are. So if they're going to be using your data, then you should be getting value for it."

Speed gets less attention in cost conversations than it deserves, Ferguson said, even though it changes how people actually work with a model. "Speed is the thing that people often don't know they can ask for," he said. "How I work with AI is so different when the responses come back as fast as I ask them."

Ferguson put a specific number on the savings from ensembling across models instead of standing up one provider: a bill that's six to eight times cheaper, and at minimum half the cost for customers running a mix of closed and open models. "We kinda guarantee that in the case where you're still using a combo of closed and open models, we can always guarantee that your bill will drop by half right away," he said. "Like we're sure about that. And then the question is can we just get you to use more stuff?"