Ankit Mathur

Ankit Mathur is a senior staff software engineer and engineering lead for AI infrastructure at Databricks. His coauthored work addresses costs that model benchmarks alone can miss: serving models through a shared interface, distributing requests with different computational demands, and measuring the expense of completing a coding task rather than buying its tokens.

Making models usable

At Databricks, Ankit coauthored the March 2023 introduction of Dolly. The team fine-tuned an existing six-billion-parameter EleutherAI model on instruction examples from Alpaca, demonstrating that an older, relatively small model could acquire substantially better instruction-following behavior. The collaborative experiment explored a more accessible route to customizable language models; it did not establish equivalence with ChatGPT.

Later that year, he coauthored the introduction of a unified foundation-model interface. Databricks Model Serving brought Databricks-hosted and external models behind a common API, with centralized permissions and monitoring. This let teams use different models through shared operational controls.

Serving requests and processing datasets

With Ahmed Bilal and Youngbin Kim, Ankit coauthored the 2025 introduction of serverless batch inference. AI Functions let users classify, translate, summarize, or extract structured information within SQL and Python workflows. A user could test a query on a few rows, refine its filters, then remove the limit to process the larger dataset. The service handled inference capacity while access to the underlying rows remained governed by the platform.

His coauthored account of Databricks’ inference architecture explains how the service distributes requests whose computational demands vary. Requests consume estimated model units, reflecting their workload rather than counting each request equally. Long inputs and long outputs impose different costs; routing and autoscaling use those estimates to distribute work and allocate capacity to customers. Request counts alone can conceal an imbalance, leaving some servers overloaded while others sit idle.

Evaluating and routing coding agents

Ankit’s coauthored work on coding agents connects infrastructure decisions to completed engineering tasks. It examines three related problems:

• Building an evaluation agents cannot shortcut. Databricks’ internal coding benchmark used reviewed changes to its own codebase and held-out tests to evaluate model-and-tool combinations. Early results exposed solution leakage: agents could recover the intended changes from Git history. The team isolated runs from that history before continuing. Its cost comparisons also challenged token prices as a purchasing shortcut: cheaper tokens could produce a more expensive completed task when an agent needed substantially more work.
• Choosing a model without wasting the session cache. In Smart Routing, Ankit and his coauthors describe a small classifier that assesses a coding task and selects its model. Simpler work can use cheaper models, while harder work receives more capable ones. Retaining that choice through the session preserves prompt-cache efficiency. The tradeoff is that an opening prompt may describe the eventual task poorly, and a reused session can outgrow its initial assignment.
• Managing access and spending across tools. Ankit also coauthored the introduction of coding-agent support in Unity Gateway, bringing access controls, audit records, usage visibility, and spending limits together across coding tools. These controls let organizations manage what agents can reach and examine what their use costs as developers adopt multiple tools.

Together, these contributions explain distinct parts of running coding agents inside an organization: testing whether they complete representative work, choosing models with attention to session behavior, and tracking access and expenditure across tools.