Agents That Own Their Inference — Du'an Lightfoot & Khaja Omer, Akamai Technologies

Du'an Lightfoot’s first agent, built in January 2023, parsed a model response, called a tool through Python, and returned the result to the model. Repeating that loop became expensive. Moving to local models introduced another problem: a smaller model could be less capable, while the GPU still had to hold the growing conversation. Lightfoot and Akamai colleague Khaja Omer begin with that practical tension—how to control cost and latency without losing the ability to finish the task.

The recurring unit of work is larger than one answer. A model chooses an action, a tool executes it, and the result returns to the model. The model then revises its plan, calls another tool, or finishes. Each pass generates tokens and carries context forward. A model that responds quickly to one prompt can still produce a slow agent if the task requires many passes.

To “own your inference” here means operating the model on a GPU you own or rent. The distinction is operational control: choosing the hardware, serving configuration, and location of the data. A cluster, data center, laptop, or private cloud network can contain the inference path when the surrounding system is configured that way.

The initial decision has several independent dimensions:

• Utilization. A steady workload can keep a dedicated GPU busy. Low or bursty traffic can leave paid capacity idle.
• Data requirements. Intellectual property, governance, or compliance requirements may favor keeping requests within infrastructure you control.
• Capability and operations. Hosted providers can supply models whose hardware requirements exceed your budget. They also absorb serving work that your team may lack the expertise to perform.

The workshop gives each participant one GPU and a Kubernetes namespace within a cluster described as having more than 200 GPU nodes. Each namespace contains a Jupyter notebook and a vLLM inference endpoint; a separate agent pod is also part of the planned environment. Participants change the serving deployment and rerun requests against it, so optimization becomes a deployment exercise rather than just a client-side prompt change.

vLLM supplies an OpenAI-compatible API and two useful serving mechanisms:

• Continuous batching. Finished requests can leave while new requests enter, instead of waiting for an entire fixed batch to finish. Lightfoot’s school-bus analogy is a bus that can pick up a new child as another gets off.
• Paged attention. KV-cache memory is managed without giving every short request a large contiguous allocation sized for a longer one. This helps the server accommodate requests of different lengths while its scheduler manages the active work.

The endpoint is only one part of a production service. In the setup described, each serving endpoint hosts one model. Multiple models therefore need additional endpoints and routing. Authentication alone also does not supply authorization or rate limiting. The presenters discuss gateways and tools such as KServe and llm-d for the surrounding routing, scaling, and access-management work.

The first connection uses Qwen3-4B through the compatible API. An early correction matters: the model has four billion parameters and starts in BF16, rather than FP8. At two bytes per parameter, its weights occupy roughly 8 GB. Precision changes the bytes used to represent the weights; it does not double the parameter count. Requests from the notebook then travel to the endpoint inside the cluster.

What does a streaming response reveal that a completed answer hides? The notebook records arriving chunks and separates time to first token from subsequent token generation. One run reports a first token after 37 milliseconds and generation around 40 tokens per second. That second number describes one request. Aggregate throughput can rise when multiple requests run together, even if each individual stream stays near that rate or slows down.

The CPU comparison is simulated because the notebook container has only 1 GB of memory; it is not a measured CPU inference run. A subsequent GPU run reports 45 tokens per second. Whether that is sufficient depends on the application: a chatbot needs a responsive first token, while a background job may tolerate slower generation if it completes the work well.

The first “agent” is deliberately modest: a system prompt assigns the role of an Akamai solutions architect, followed by a single request. It has no tools yet. Adding tool calls changes both the latency calculation and the context budget, because the system prompt, prior responses, and tool results become part of later model inputs.

Server metrics and agent traces answer different questions. The workshop’s foundation.py helper reads vLLM metrics through Prometheus; Grafana can display how the server behaves under load. Lightfoot also recommends tracing agent steps with a tool such as Langfuse, which can run beside the agent in the cluster. Those traces reveal token counts and latency for individual tool calls. If a smaller model repeatedly calls tools and takes longer to finish, the useful correction may be a clearer prompt, fewer tools, or a narrower role.

Increasing prompt length makes the first token take longer in the notebook exercise. Lightfoot directs attention to the slow requests rather than relying only on the median. That matters for agents whose later turns accumulate substantially more context than their opening request: a quick first exchange can conceal the expensive part of the task.

The cost exercise compares an illustrative workload of 200,000 tokens per day: about $1,500 per month through a hosted service versus about $1,000 for a rented GPU under the notebook’s assumptions. These are scenario figures, not a general break-even price; the provider rate, GPU size, model capability, and actual workload determine the comparison. Runtime choice also follows workload: the presenters point to SGLang for shared long prefixes and multi-turn requests, TensorRT for NVIDIA throughput, and llama.cpp for heavily quantized edge deployments.

GPU memory must accommodate both the model and the requests it serves. The workshop starts with roughly 8 GB of BF16 weights for the four-billion-parameter model, then considers the remaining room for KV cache. The prompt establishes an initial cached context; generated tokens extend it. System instructions, tool calls, and tool results all contribute to the context the application needs to support.

Model files explain what the server is loading. The tensor files contain weights, while config.json describes the architecture the runtime must interpret, including layers, attention heads, and vocabulary. These architecture fields should not be treated like serving knobs: changing them without matching weights can prevent loading. Fine-tuning or LoRA adapts behavior through a different mechanism.

The useful weight calculation is simple: parameter count multiplied by bytes per weight. BF16 uses two bytes; FP8 uses one. A uniformly represented four-billion-parameter model therefore has about 8 GB or 4 GB of weights respectively, before considering other memory needs. The workshop uses that arithmetic alongside memory bandwidth to estimate a generation ceiling, then asks whether the intended concurrency and context lengths fit.

How does that become a user-capacity estimate? In the notebook’s example, the available KV pool holds about 37,000 tokens after accounting for the weights. Four simultaneous users with 8,000 tokens each consume 32,000 tokens of that pool. Each user’s allowance must cover the relevant input, output, and tool context together. The pool is shared across active requests; it is not a separate 37,000-token allowance for everyone.

Lightfoot connects this to a previous workshop: could roughly three or four 20 GB GPUs serve 80 students? Counting students alone could not answer it. The required context per student, simultaneous arrivals, batching behavior, and slowdown under load all had to be evaluated. The memory estimate narrows the possible deployment; a realistic workload test checks how it behaves.

Prefill and decode put different work on the GPU. Prefill processes the supplied prompt and contributes to the delay before the first streamed token. Decode generates the following tokens. Measuring them separately makes a long-prompt problem distinguishable from slow ongoing generation. In one notebook run, the first-token measurement is 57 milliseconds and the reported decode interval is 25 milliseconds.

The KV-cache exercise compares repeated attention work with retaining information from the context for reuse. The practical point is reuse during generation, rather than learning new model weights from a prompt. The notebook works through attention calculations, then grows a request while observing peak KV utilization and utilization after completion. This connects the abstract context budget to memory occupied during an actual request.

What changes when a request can reuse an already processed prefix? The cold-versus-warm exercise reports a drop from 220 milliseconds to 41 milliseconds. Prefix caching retains reusable KV work for the shared beginning of requests. Lightfoot distinguishes this serving mechanism from hosted providers’ prompt-cache billing discounts: both can reward repeated context, but one measurement concerns local serving latency and the other concerns the provider’s pricing policy.

A dense model uses its full set of layers to produce each token. A mixture-of-experts model activates selected experts, so its total parameter count can be much larger than the portion used for an individual token. The workshop’s illustrative 30-billion-parameter configuration has 128 experts with eight active per token. That distinction can reduce work and weight movement during generation, even though the complete model still presents a larger storage requirement.

The bandwidth calculation gives the four-billion-parameter baseline an estimated ceiling around 45 tokens per second, close to the observed rate around 40. That ceiling belongs to the model representation and hardware assumptions being calculated. It does not mean every serving configuration must produce the same result: the later quantization exercise changes how many bytes the GPU must move, while batching changes aggregate work across requests.

Thinking introduces another budget: how many tokens the model generates before finishing. The demonstrated response uses 35 tokens with thinking disabled and about 600 with thinking enabled. Extra reasoning may help the model interpret tool results or troubleshoot, but it consumes generation time and context. The presenters confirm that thinking budgets can be adjusted, making the useful question how much reasoning a task needs rather than whether more tokens are always better.

This foundation ends by separating the resource pressures: prefill is presented as compute-heavy, while decode is often constrained by memory bandwidth. The workshop discusses separating prefill and decode pools conceptually, but does not deploy that arrangement. Its hands-on optimization continues on the single-GPU serving endpoint.

Omer begins optimization with quantization. In the 20 GB memory budget used for this exercise, smaller weights help in two ways: they leave more space for KV cache, allowing more requests to remain active, and they reduce weight movement during decode. Those are separate benefits—capacity for parallel requests and speed within generation.

The baseline load test uses approximately 256 input tokens and 128 output tokens per request across concurrency levels. With BF16, one request generates 37.5 tokens per second, with a 61-millisecond time to first token. At concurrency 32, aggregate throughput reaches 714 tokens per second. Moving from concurrency 32 to 64 raises first-token latency from 277 to 1,300 milliseconds. Higher throughput has already begun to carry a substantial waiting-time cost.

Quality checks have to accompany that performance baseline. Omer expects BF16-to-FP8 conversion to preserve useful quality for many tasks, but treats that as something to test on the actual workload. A concrete failure case is malformed JSON: text can look plausible while a function call fails because its structured arguments are invalid. Hallucinations, broken reasoning, and repeated loops are other regressions an application-specific evaluation should catch.

The deployment change happens in vllm.yaml. A persistent volume holds downloaded models, and the deployment arguments select which model vLLM loads. Omer switches from the BF16 model to a pre-quantized FP8 version from Red Hat, checks the diff, and applies the deployment. The first rollout is delayed; an audience member spots duplicate model flags, which Omer corrects before rerunning it. A valid manifest and a successful rollout are necessary before attributing a result to quantization.

The format must also suit the hardware. Omer distinguishes a GPU’s ability to support a representation from its ability to accelerate the corresponding calculations. A smaller representation can disappoint if the hardware cannot execute it efficiently. Mixed-precision recipes further complicate the simple expectation that every weight shrinks by the same factor: the selected model is described as retaining some components at higher precision.

How much does the successful FP8 rollout change this workload? Single-request generation rises from 37.5 to 53.4 tokens per second. At concurrency 64, aggregate throughput rises from about 800 to 1,140 tokens per second—roughly 1.42 times the baseline. First-token latency still rises sharply between concurrency 32 and 64. The simple quality cases pass, which supports this workshop example without establishing equivalent quality for every agent task.

Speculative decoding changes how candidate tokens reach the main model. A smaller draft model proposes several tokens ahead of time, and the target model verifies them. Omer’s example proposes four tokens: the verifier accepts the first two and discards the remaining two. The intended gain comes from accepting useful work from a cheap drafter instead of making the target generate every token through a separate sequential step.

Where does proposed work survive, and where is it wasted? The branch below follows the four-token teaching example. Acceptance determines how much of the draft becomes usable output; discarded tokens still consumed drafting resources. The example is a mental model of acceptance, rather than a complete account of the verifier’s sampling algorithm.

Draft length is therefore a consequential setting. Proposing four tokens and accepting two yields 50% acceptance in the example. That can waste enough drafting work to erase the expected benefit, although acceptance percentage alone does not determine net speed. Omer recommends testing shorter and longer drafts; his separate home-lab experience with a diffusion drafter favored eight tokens, with worse acceptance beyond that. The right length depends on the method and workload.

The discussion also distinguishes ways to obtain draft proposals: a separate small language model, a drafter trained alongside the target, or a diffusion drafter that proposes multiple tokens together. Before configuring the experiment, an audience question returns to quantization. Omer notes that the selected FP8 model is about 5.1 GB rather than 4 GB and describes selective higher-precision components as the reason. The practical lesson is to use the loaded model’s actual footprint rather than assume a uniform twofold reduction.

The experiment establishes a fresh FP8 baseline, then sets vLLM’s speculative configuration to a 0.6-billion-parameter Qwen drafter and four draft tokens. Another audience question identifies GPU memory utilization set to 0.7. Reserving headroom can make sense when deployments share a GPU, but Omer says this example has no particular need for that reservation. Too little allocation leaves capacity unused; too much can cause out-of-memory failure.

The fresh FP8 baseline produces 58.2 tokens per second at concurrency one, with a reported 42-millisecond latency. Enabling speculative decoding drops generation to about 16 tokens per second and increases time to first token. The change is a slowdown, not the expected acceleration.

Does concurrency rescue the configuration? At concurrency 16, aggregate throughput falls from about 700 to 240 tokens per second. The regression therefore affects both an individual stream and the server’s combined output. The load-test comparison is useful precisely because a promising mechanism has failed under the actual deployment.

Several mechanisms could explain the loss:

• Rejected draft work. Low acceptance means paying to generate candidates the target discards.
• Draft length. Four tokens may be too ambitious for this pairing; shorter drafts could reduce wasted work.
• Scheduling and contention. Two models share the GPU, and the serving scheduler must coordinate the additional work.
• Model compatibility. A drafter must predict continuations that the target will accept for the intended task.

Omer attributes this result primarily to an unsuitable drafter-target pairing and its training or quantization recipe, saying the GPU has sufficient headroom. The recording does not show a numerical acceptance-rate breakdown or a controlled isolation of those causes, so that diagnosis remains narrower than the measured slowdown. The operational decision is clear: comment out speculative decoding and redeploy the baseline.

The questions afterward sharpen the pairing problem. Precision differences can change how well a drafter’s proposals match the target, and a drafter useful for code may fare differently on academic writing. Omer recommends trying model-associated drafters and evaluating the intended domain. A larger drafter might improve predictions while consuming resources the target needs; the goal is a sufficiently capable small drafter, not a universal parameter-count rule.

Why can verification be faster at all? Omer’s explanation is that proposed positions can be evaluated together rather than discovered one at a time through sequential generation. He also describes distillation from target-generated data and training alongside the target as ways to produce a drafter. His separate home-lab report of roughly two- to threefold gains illustrates that successful pairings exist, but it does not change the workshop result: this pairing should be removed.

The next exercise asks how much concurrency the deployment should accept. Throughput initially rises as more requests keep the GPU occupied. Eventually it flattens, while latency can continue increasing. Omer calls the useful operating region the “knee”: near the throughput plateau, with waiting time still acceptable. Keeping the GPU busy only helps if the service still meets its latency requirements.

While the test runs, speculative decoding returns as a workload question. Omer reports roughly 1.5-fold improvement for browser use and closer to threefold for coding in his own experience. Browser actions still take time outside inference, so faster token generation is only one contributor to a faster agent. When asked for benchmarks, his response is blunt: “I never get the same results.” The practical recommendation is to measure acceptance and performance on the tasks the agent actually performs.

The discussion broadens from the fixed four-token setting to adaptive draft lengths that respond to acceptance and GPU load. Omer does not know whether the proprietary systems raised by the audience use the technique. These are separate questions: a serving method can be available for an open deployment without establishing how a hosted provider implements its own inference.

Then the notebook hangs. Refreshing and opening another tab do not restore normal input, although scrolling still works. Omer jokes that the cluster has taken “a vacation,” explains the intended knee around concurrency 32, and switches to a working notebook environment. A later cell needs adjustment because the exercise was designed to continue from the previous notebook. The cause of the failure is not established; the important consequence is that the experiment’s environment has changed.

The tuning controls address different constraints:

• GPU memory utilization. Allocate enough memory for this server while preserving room for other workloads and avoiding out-of-memory failure. Omer suggests testing 0.9 or 0.95 for a GPU devoted to one model.
• Maximum sequences. Limit how many requests vLLM processes at once. The deployment starts at 32.
• Model length. Set the context limit to match the application’s needs.
• Batched tokens. Adjust how much token work is scheduled together, with prefill performance and GPU capacity in mind.

Omer recommends changing one flag at a time and repeating the benchmark. A concurrency knee is evidence of a bottleneck, but it does not by itself prove the hardware is exhausted: a configured scheduling limit can also impose one. Once measurements approach the estimated hardware ceiling, more or better hardware becomes the next option. Until then, the task is to identify which constraint is responsible.

The runtime question returns near the end: should a personal assistant use Ollama or vLLM? Omer recommends starting with hardware and concurrency. Ollama or llama.cpp can make model exploration convenient, and one sequential agent may be adequately served that way. Once a model is chosen and something like 20 agents must run in parallel, vLLM or SGLang offers serving controls intended for higher concurrency. The required workload earns the additional operational complexity.

The final concrete change raises the maximum sequence count from 32 to 128. With a 32-sequence limit, offering 64 concurrent requests does not make all 64 active at once. Increasing the limit gives the scheduler room to process more requests together, provided memory and compute can support them. Omer applies the deployment and repeats the load test.

During the rollout, an audience member asks about bursts that remain after tuning, including possible spillover to serverless. Omer discusses autoscaling rather than demonstrating a hosted fallback: additional GPU capacity can follow demand, and traffic-history models can inform predictive scaling. That addresses capacity over time; the sequence setting addresses how the current server schedules work.

At concurrency 64, the final comparison reports throughput rising from about 800 to 1,100 tokens per second, while the reported latency drops from about 2,000 to 800 milliseconds. These are the two outcomes to inspect together: more combined output and less waiting. Raising a scheduling limit can improve both when the earlier limit prevented useful parallel work.

Which baseline does that result belong to? An audience member notices the single-request rate, and Omer explains that the replacement notebook is now running the full-precision model rather than FP8. The move from 50-something to 30-something tokens per second therefore includes a model-precision change. The final serving comparison should be read within this recovered environment, rather than as a continuation of the earlier FP8 experiment or a cumulative gain from every optimization.

The session closes with the workshop code and presentation offered for reuse. Its strongest practical lesson is earned by the deployments themselves: FP8 improves the measured workload, speculative decoding makes it worse and is removed, and a higher sequence limit improves the final concurrency test. Owning inference gives the operator these choices—and the responsibility to keep the model, workload, latency, throughput, and output quality visible when making them.