When Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec

Steven Latré leads AI at imec, the Belgian semiconductor R&D center. His group studies how AI workloads behave on hardware, and imec’s own use of coding agents gave it a practical reason to investigate. In 2024 and 2025, the team wanted access to the best frontier model it could get. As pricing shifted from subscriptions toward metered tokens, it started routing tasks to models suited to their difficulty. Token consumption kept growing, though, and a different question came up. What would it cost to run those tasks on GPUs that imec leased or bought?

With an API, the team could pick a model, measure whether it did the job, and read the cost off the bill. Hosting an open-weight model brought a series of decisions that the provider had previously made. Which inference engine should serve it? How much GPU memory would it need? What balance of CPUs and GPUs would keep the system busy? Even the choice of coding harness, the software that drives the model through a task, could change the answer.

The team chose long-running coding tasks because Latré saw them as an early candidate for bringing inference in-house. A coding agent can occupy a machine over an extended job. An interactive application has a different demand pattern and may have different economics. The question for the benchmark was how much infrastructure it would take to deliver a given level of performance on the coding work itself.

Using SWE-Bench Pro, imec ran thousands of trials across three coding harnesses, Codex, Pi, and Claude Code. The researchers compared a conventional API setup with open-weight models of different sizes. They matched each hosted model to hardware intended to be sufficient for multiple simultaneous users. For each configuration, they tracked the fraction of tasks the agent resolved, the latency, and the cost. That let them follow a model’s benchmark score all the way through to the machine needed to achieve it.

The models fell into three rough groups. Qwen and DeepSeek, on the smaller end of this study at up to a few hundred billion parameters, resolved roughly 35–40% of tasks. GLM reached what Latré described as Opus-level performance. Kimi K3 resolved 86%. Latré suspected that some benchmark material might have appeared in Kimi’s training data, a qualification on how far to generalize its lead.

Adding rental prices changed the picture. Qwen and DeepSeek delivered similar resolve rates, yet the cost of renting their respective GPU configurations differed by a factor of four. Moving from GLM to Kimi K3 produced the opposite pattern. Latré reported an accuracy improvement of about 20% for an increase of only 15% in GPU rental cost.

Hardware comes in steps. A modest change in model choice can move a deployment into a different class of machine and produce a large jump in price. Elsewhere on the same curve, a considerably stronger model can fit on hardware that costs only a little more. The rental price buys the capacity of that machine. How much work fits on it is another measurement.

A model that fits in GPU memory still needs room for its users. Alongside the model weights, the server holds a key-value cache for each active session so the model can reuse attention computations from earlier tokens. As more agents work at once, their caches compete for the memory left over after loading the weights. There is also a latency limit. A machine may have room for another session even when the resulting slowdown makes it impractical to add one.

Imec measured how many simultaneous users each configuration could support reliably. Comfortable concurrency fell sharply for the larger models, even though the team gave them larger machines. All the self-hosted configurations were slower than the Claude Code baseline. Kimi K3, the model with the highest resolve rate, supported only about four concurrent users on its tested configuration and still took about three times as long as the baseline. For a team with several coding jobs arriving together, those figures matter as much as the chance that any one job succeeds.

Imec compared three ways of paying for the work: API usage, rented GPUs, and purchased on-premises infrastructure. Utilization connected the token bill to the hardware bill. At the lower model and hardware tiers, the GPUs had to run at capacity about 89% of the time for the move away from the API to break even. Latré considered that infeasible in practice. A small machine may be cheaper to acquire, but the API work it replaces is also relatively inexpensive.

For the tested system with eight B200 GPUs, the break-even figure dropped to about 15% utilization, which Latré considered realistic. Across the configurations, he identified GLM-class coding workloads as the point where moving off the API started to look attractive. Imec’s total-cost-of-ownership analysis put the savings for this coding-assistance application at roughly 24–25%. Applying that estimate elsewhere would require knowing which API work the machine would replace and how often there would be enough of it to keep the GPUs occupied.

Viewed through resolve rate alone, the harness comparison looked uneventful. Across the benchmark, Codex, Pi, and Claude Code achieved broadly similar task accuracy for a given model. But when imec counted the tokens generated on the way to a solution, the harnesses diverged. With GLM, Codex generated about half as many tokens as the other harnesses.

That reduction showed up in elapsed time on the GPUs. Codex occupied them for about half as long and incurred about half the GPU usage cost for comparable task performance. On rented hardware, the difference is billable time. On a machine the team already owns, it is time available for another coding job. Both effects are easy to miss in a table that reports only how many tasks passed.

The study followed one application. An enterprise sharing infrastructure across several applications would have to measure how their demands combine. Quantization and inference optimizations also leave room for the results to change. The team used speculative decoding, for example, but did not explore its configuration in detail.

Latré described plans to turn imec’s benchmarking pipeline into a public dataset that could track new models and hardware as they appeared. The center was also forming an alliance connecting software teams with chip and infrastructure providers. For both groups, the unit worth tracking is a completed task under a specified load. It preserves the connection between what a model can solve and what it costs to put that capability in users’ hands.