20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna AI

What makes an AI model state of the art for an application? Whether the goal is research performance or a deployed product, two familiar routes promise an answer: consult a public leaderboard or evaluate candidates internally. Used naively, both can lead to the same decision—choose one large foundation model because it has the highest measured quality. The missing question is what that measurement establishes about the application you actually intend to build.

For image editing, the shortcut is straightforward: open Design Arena and select ChatGPT image, the leader in the snapshot Bertrand Charpentier presents. That gives you a reasonable first candidate with little effort. It does not yet tell you whether the model suits your users, their editing requests, or the conditions under which they will use it.

Put Arena, formerly LM Arena, beside Design Arena and Artificial Analysis, and the image-editing rankings disagree. The top entry changes, but so do the relative positions of models further down the list. In the displayed comparison, Hunyuan moves from tenth on Artificial Analysis to fifth on Arena. These are the talk’s leaderboard snapshots, not current standings.

The comparison is also complicated by duplicate entries and models that appear on one board but not another. Even the numerical ratings are not interchangeable: Charpentier points to an Elo range of roughly 1,100–1,300 on some boards and different ranges elsewhere. An Elo value only becomes useful with the context of the comparison that produced it; placing numbers from separate boards side by side does not establish a common quality scale.

Several leaderboards therefore provide a better starting point than one. Charpentier interprets substantial rank disagreement as a reason to consider whether candidates are approximately equivalent, rather than treating one board’s winner as universally superior. Disagreement is a reason to investigate the evaluation context, not proof that every model performs equally well.

An aggregate image-editing score combines different capabilities: removing an object, changing a background, editing text, and more. Split those tasks apart and the rankings change again. ChatGPT image is not first in any of the task-specific rankings shown. Different training emphases can produce different strengths, so a product built around object removal should start with object-removal results rather than an overall editing winner. The application determines which comparison matters.

Next, ask what the evaluation sampled. Charpentier describes the Artificial Analysis examples as using a few thousand samples per model, compared with millions of daily inferences for some Pruna models. That contrast motivates collecting evidence from the actual workload. Traffic volume alone does not make an evaluation representative: the prompts and operating conditions still need to resemble the application.

Preference rankings are built from battles: users see two outputs and choose the one they prefer. Charpentier reports that most models in the comparison lose at least 40% of their battles. That is a relative preference loss rate, not an application failure rate. A model can lead overall while another model is preferred for the subset of requests your users make. More samples help when they cover that subset and reproduce the final use-case conditions.

Internal evaluation seems to offer more control. In image and video generation, it often means trying a few prompts on several models and inspecting the outputs. Alternatively, a team runs an automated benchmark and chooses the highest scorer. Both approaches still depend on what the evaluator asks and which examples enter the test.

Charpentier tests manual inspection with three images and asks the audience to choose a favorite. Before voting, someone asks what the requirement was; another question asks for the prompt. These questions separate two possible judgments: which image looks preferable, and which image follows the instruction. Charpentier recalls the prompt approximately as a little guy and a parrot. Audience members choose different outputs, immediately exposing the limits of one evaluator’s taste.

The exercise repeats with a prompt he recalls approximately as a man eating soup with pasta. Some people switch their preferred image position. The model positions have stayed fixed: Seedream on the left, FLUX.1 in the middle, and Pruna’s P-Image on the right. A preference formed from the first example does not necessarily survive the second.

Manual inspection introduces two distinct biases:

• Evaluator bias: your visual preferences need not match those of other users.
• Sample bias: a few selected prompts need not represent the requests on which a model will be judged.

Inspection remains useful for developing intuition. Human evaluation becomes stronger evidence when it includes many evaluators and a broader set of relevant examples.

Automation does not remove the need to interpret the test. Charpentier compares eight models using CLIP-score evaluations across different datasets. Their order changes across the displayed comparisons, while the numerical gaps remain small. He describes the displayed scales as 0–1 or 0–100; those scales should not be treated as a universal normalization for CLIPScore. The practical problem is that a small difference in a broad compatibility score may not resolve which model is better for the intended capability.

Start by understanding what each metric measures, then use more than one. If the application needs readable text inside generated images, text-rendering evaluations address that capability more directly. In the displayed text-rendering evaluations, Z-Image consistently ranks first and P-Image second. The differences are also clearer on the stated 0–1 scales. That visible separation is useful evidence, though it is not itself a statistical significance test.

Running a benchmark is not the same as understanding its result. A score is useful when you can explain which capability it measures and why that capability matters to the product. Otherwise, choosing the highest number simply relocates the original model-selection problem into a metric.

Public rankings and internal benchmarks share another limitation when they emphasize quality alone: quality can be purchased with more computation. Charpentier illustrates the cost using an approximately 26,000-image evaluation workload for ChatGPT image, at 62 seconds per image. He is uncertain which leaderboard supplied the battle count, so this is best understood as his workload estimate rather than a reconstruction of a leaderboard’s execution. A battle compares outputs; it does not automatically imply one distinct image generation.

For that workload, Charpentier estimates about 20 days of accumulated generation time, about $5,000 in cost, and 556 kWh of energy. Multiplying the rounded count by the per-image latency gives approximately 18.7 days, consistent with his rounded time estimate. This sums per-image latency; it does not establish wall-clock benchmark duration or GPU-hours. His energy analogy comes from his own Strava estimate: roughly 400 marathons. It is an illustration of scale, not a standardized energy conversion.

He then presents a Pruna alternative described as generating and editing images in under one second. The comparison below preserves his estimates for the same evaluation count; it does not imply that the outputs have equal quality or that these are universal model costs.

The alternative’s exact model and measurement configuration are not specified here. The useful decision is whether any additional quality justifies the extra computation, money, energy, and waiting time.

A Pareto frontier makes this decision visible without collapsing everything into one score. Put an efficiency measure on the horizontal axis—generation latency or price per image—and a quality measure, such as Elo, on the vertical axis. Lower latency or price is better; higher quality is better. A model is dominated if another candidate is at least as good on both objectives and better on one. The frontier contains the remaining choices: improving one plotted objective requires sacrificing the other.

The red frontiers in Charpentier’s plots contain roughly three or four models. Their quality scores cluster around 1,100–1,200, while efficiency varies substantially; he gives roughly twentyfold faster generation as an illustrative difference between model choices. That does not make every frontier model interchangeable. It means there are several defensible choices, depending on how much the application values the quality gain relative to latency or price.

The next improvement is to make the vertical axis specific to the task. For text rendering, plot text-rendering quality against efficiency instead of a general capability score. Charpentier describes optimizing FLUX.2 and other FLUX models with BFL, a collaboration also described in Pruna’s FLUX.2 optimization report. In the text-rendering comparison he presents, optimized models run faster while remaining on the frontier. Optimization therefore changes which tradeoffs are available, not just which existing model you select.

Benchmarking remains useful when it reflects the application: enough relevant samples, realistic use conditions, multiple meaningful measurements, and explicit efficiency costs. Charpentier argues that this process can favor smaller models tuned to particular preferences rather than automatically selecting a large general foundation model. The result is a choice grounded in the workload instead of a universal winner.

Pruna’s deployment route is to serve what it calls preference models behind endpoints. Charpentier advertises image and video models with runtimes of roughly one to five seconds. For teams optimizing their own models, he points to the open-source Pruna package, alongside efficiency research materials and a course. These offer two routes to the same goal: use an already optimized model or apply optimization to the model your application needs.

The closing audience question asks how compression works for image and video models. Charpentier first names two ways to change the model’s components:

• Quantization: choose different quantization treatments for individual modules rather than applying one treatment uniformly.
• Pruning: remove components judged unimportant.

Both target the work performed inside the model. The next opportunity concerns how often that work repeats.

For image and video generation, Charpentier describes a typical process with roughly 20–50 denoising steps. The same backbone is used repeatedly to generate the content, making repeated denoiser computation a substantial optimization target. He identifies distillation and caching as ways to reduce that work, illustrating a reduction from 50 backbone computations to 20 or even four, depending on how aggressive the optimization is. Those counts are examples of the opportunity, not guaranteed results for every model.

The questioner already uses caching for video and asks what else could make inference faster. Charpentier distinguishes the caching algorithms available in Pruna’s open-source package from internal algorithms used by models served behind its endpoints. He does not introduce a specific new algorithm or implementation in the answer. The practical distinction is between optimizations a team can apply through the public package and those available through the behavior of a hosted model.