The Rise of Open Models in the Enterprise

Does slow enterprise adoption mean AI is mostly hype? The question matters because enterprises have enormous reach and spending power. If they cannot put AI to work, the economic shift promised by the technology will take longer to materialize.

Amir Haghighat, Baseten’s co-founder and CTO, approaches that question through conversations with more than 100 enterprises over the preceding two years, ranging from public software companies to Fortune 50 soft-drink businesses. This is a commercial vantage point, not a representative market survey. It also has a particular bias: Baseten sells horizontal inference infrastructure rather than a finished application for one business function.

Enterprises were already buying AI products for sales, marketing and customer service. Buying AI applications and building with AI are different forms of adoption. The latter creates room for companies to turn their own knowledge and processes into software. Haghighat compares this with the earlier enterprise software expansion: if companies had only bought Salesforce-style applications, rather than building technology themselves, the surrounding market for Snowflake, Databricks and Datadog would have looked very different.

The usual starting point in these conversations was OpenAI or Anthropic. Their APIs made it easy to begin, while enterprise procurement brought cloud deployments through Azure or AWS into the picture for security and privacy. Existing predictive ML teams often became the teams building AI applications. Once that arrangement worked, convenience and organizational inertia both favored keeping it.

In 2023, the CIO of a large insurance company described a dedicated deployment of GPT-4 or GPT-3 as something engineers could “toy around with.” The deployment existed, but the assignment was exploratory. In 2024, Haghighat estimates that 40–50 of roughly 100 enterprise contacts had something in production. By 2025, his conversations had shifted again: teams were encountering reasons they might not be able to build indefinitely on closed frontier models alone. This was a progression from experimentation to operational constraints, not a wholesale rejection of closed models.

Vendor lock-in was not a frequent complaint in Haghighat’s conversations. With OpenAI, Anthropic and Google available, customers saw alternatives and some interoperability. Moving between them still meant rerunning evaluations and adjusting prompts. That is a narrower claim than universal API compatibility: current Claude documentation describes a limited OpenAI SDK compatibility layer, not a complete common contract across providers.

Cost had also seemed less urgent in the earlier conversations because customers expected falling token prices to solve it. Compliance, privacy and security were similarly perceived as addressed through arrangements with model vendors and cloud providers. Haghighat describes dedicated deployments inside existing VPCs, but private access does not necessarily mean the model runs in the customer’s VPC: AWS’s PrivateLink explanation distinguishes a customer VPC from the Bedrock service account. The deployment details matter; the observation here is that these customers considered their requirements satisfied.

A better model for a particular task need not be a better general model. The enterprises Haghighat describes were not trying to outbuild OpenAI at general intelligence. They were finding tasks for which a frontier model was not the right tool.

Large health plans provide one example. They process millions of medical documents, including prior authorizations and claims, and need to extract CPT procedure codes, diagnosis codes and prescriptions. Simply sending those documents to Claude or a GPT model did not meet the needs of the cases Haghighat encountered. The health plans also possessed years of labeled data. That gave them a basis for improving the specific extraction task, and he reports that they did improve it, without supplying benchmark scores.

Medical transcription creates a related opportunity. Understanding specialized medical jargon can be more valuable than broad, generic transcription capability. Bringing that model work in-house lets the organization target the vocabulary and errors that matter in its own setting. The competitive asset is the combination of a narrowly defined task and relevant enterprise data.

The next pressure is latency. Haghighat characterizes shared model APIs—including services hosting open models—as optimized for throughput and queries per second, sometimes at the expense of individual response latency. An AI phone call exposes that tradeoff immediately: the user experiences the pause before the system starts answering.

For voice interactions, serving choices must reflect the latter two requirements. A generally capable model behind a convenient endpoint does not, by itself, establish that the application will respond quickly enough.

The expectation that falling token prices would resolve cost concerns ran into agentic workloads in 2025. Haghighat reports seeing a single user action produce 50 inference calls in an agentic application. That is an observed example, not a universal agent workload ratio, but it changes the unit of analysis: the business pays for all the inference needed to complete an action, not just one inexpensive call.

The relevant accounting relationship is:

As calls multiply, teams must bring the total back within the economics of the product and demonstrate ROI. Running models on paid compute can give them more control than accepting a provider’s per-token prices and margins. Haghighat reports cases where this approach was cheaper; the move creates an opportunity to improve economics, not an automatic savings guarantee.

A separate strategic concern follows: if competitors all use the same frontier models, where does differentiation come from? Some CIOs and CTOs want control over the AI layer itself, in addition to their workflows and applications. Customizing models becomes a way to express an enterprise’s particular capabilities, rather than relying entirely on capabilities available through the same APIs to everyone else.

Adopting open models makes iteration, fine-tuning and distillation possible, but it also changes what the organization must operate. A simple API dependency becomes inference infrastructure that must scale. The team still needs to deliver applications quickly; a long hiring cycle followed by a long infrastructure build can consume the benefit of taking control.

A common proposed recipe is straightforward: choose an open model, run it with vLLM, SGLang or TensorRT LLM, and supply GPUs. Enterprises may have those GPUs in a data center; startups may rent them in a cloud. Either way, running inference is not the same as operating mission-critical inference. The engine is one component of a service whose performance and operational behavior still need engineering.

Latency work happens at both the model and infrastructure levels. At the model level, even choosing a speculative decoding approach opens several paths:

• Draft model: Use a separate model to propose candidate continuations.
• Medusa heads: Use additional decoding heads to propose candidates.
• EAGLE-3: Consider another speculative acceleration approach.
• MTP: Consider multi-token prediction where the model and serving stack support it.

These are alternatives to investigate, not interchangeable switches.

Haghighat points to EAGLE-3 as recent research already making a meaningful production contribution in his experience. The paper was first submitted on March 3, 2025; his approximate six-month age estimate should not be treated as its publication chronology. The larger engineering issue is the pace of change: organizations need people who can evaluate new techniques and make them work for their workload, rather than assuming an engine option delivers the result on its own.

Other optimizations cross into infrastructure. Prefix caching reuses work on shared prompt prefixes, while disaggregated serving separates serving stages such as prompt processing and token generation. Large agent prompts often contain similar material from call to call, making these choices relevant to the time before an answer starts. The target is not merely a good typical time to first token, but a reliable P99—the latency boundary that covers 99% of requests. Haghighat identifies these as engineering requirements without walking through a particular implementation.

For mission-critical inference, Haghighat poses four-nines availability, or 99.99%, as a goal. He argues that the basic model-plus-engine-plus-GPUs recipe does not guarantee more than two nines; that is his warning about an incomplete architecture, not a measured ceiling for serving engines.

Hardware fails, and serving processes can crash. Haghighat describes firsthand experience with vLLM and Triton crashes that left users waiting while services recovered. Recovery time therefore affects both availability and tail latency. Keeping excessive spare capacity might cushion failures, but it can also undo the unit economics that motivated operating open models in the first place.

Traffic bursts expose a different delay: adding capacity. One large soft-drink enterprise told Haghighat that starting another replica of the same model took eight minutes. He considers that plausible once the startup work is added together, but unacceptable for responding to a sudden burst. Requests accumulate while the new replica starts, driving up tail latency precisely when more capacity is needed.

The remaining platform work also determines whether engineers can move quickly:

• Tooling and lifecycle management: Support the ongoing work of delivering and operating models.
• Observability: Make the service understandable beyond merely emitting logs and metrics.
• Controls and audits: Meet the operational and governance requirements enterprises bring to deployment.

Performance, recovery and scaling are therefore part of a larger platform obligation, not the entire obligation.

Once these requirements become concrete—sometimes only after a team has built the stack itself—the organization faces a build-or-buy decision for inference infrastructure. Haghighat advocates buying that layer and explicitly acknowledges his commercial interest: selling the infrastructure platform is his job. Buying it is still compatible with building the enterprise’s own AI capabilities. The application, workflow and model customization can remain areas of internal investment while an infrastructure supplier carries part of the operating burden.

The closing invitation extends that decision to vendors selling into enterprises. Alongside offering to discuss inference problems with enterprise teams, Haghighat offers startups lessons from Baseten’s own right and wrong product decisions. Those choices affect whether a product can later be deployed into a customer’s own cloud. Customer-cloud deployment can become another substantial infrastructure problem; its feasibility depends on decisions made before the enterprise sale arrives.