Harshul Jain

Harshul Jain builds data and AI infrastructure and is the creator of AgentShip, a project that supplies the production machinery surrounding AI agents. His work connects ML and data platforms with a practical concern: helping engineers spend more time on application behavior and less time rebuilding the systems needed to run it.

His professional background includes work as a Software Development Engineer II in Audible’s Fraud Engineering and Content Data Science Engineering teams. He helped build systems protecting content from abuse and AWS data pipelines supporting recommendations through relationships among topics, genres, and series. His broader infrastructure work includes streaming pipelines, feature stores, and generative-AI platforms. By the time of his recorded LLM inference workshop with Tanmay Sah, he was a senior software engineer at Audible and described five years of experience building ML and data platforms.

From production plumbing to AgentShip

Jain created AgentShip after repeatedly finding that production plumbing took far longer than writing the agent logic. API serving, streaming, execution, and observability required substantial work around the agent itself. AgentShip grew out of that imbalance: shared infrastructure could make those recurring responsibilities easier to handle.

The project separates agent definitions from execution machinery through engine adapters around a vendor-independent core. Developers can describe agents and teams in YAML or Python while using shared infrastructure built with LangGraph, LiteLLM, MCP adapters, and OpenTelemetry. This separation lets developers define what an agent should do while the surrounding runtime handles how that work executes. Keeping those concerns distinct aims to make runtime changes manageable without rewriting the application.

Jain also develops LLM Inference at Scale, a public practitioner’s handbook connecting GPU memory, capacity planning, model optimization, cache management, serving engines, and production operations. He began assembling it while learning inference because useful resources were scattered. With Tanmay Sah, he developed workshop material that pairs explanations with slides, notebooks, and benchmarks engineers can inspect.

His educational work extends to The Engineer’s Digest, which he co-writes with Tanya Sah. They explain system-design fundamentals through ML platforms, databases, memory, and AI infrastructure. Jain also mentors engineers through Codementor and Topmate and reviews technical books for Manning.

Making inference choices concrete

Jain approaches inference through the constraints an application must meet. His work on inference capacity and serving starts with model weights and the GPU memory left for request state. The KV cache stores attention keys and values so generation can reuse earlier computation. Longer contexts and more concurrent requests both increase its memory requirements; latency targets impose another constraint on how much work a GPU can serve at once.

Several practical distinctions shape that approach:

• Capacity depends on the application. An interactive chat application may prioritize context and responsiveness, while asynchronous tasks can tolerate more delay to gain throughput. Hourly GPU price is therefore an incomplete measure of cost. Jain emphasizes calculating how many useful tokens the hardware can serve under the application’s context, concurrency, and latency requirements.

• Prefill and decode expose different bottlenecks. Prefill processes the incoming prompt and builds its attention state, involving substantial parallel computation. Decode generates successive tokens and often depends heavily on moving model weights and cached state through GPU memory. Time to first token and inter-token latency reveal different kinds of work. Understanding the limiting resource helps engineers choose an optimization that addresses the bottleneck they actually face.

• Cache and scheduling techniques solve different problems. Paged attention allocates cache memory in blocks as requests grow, reducing waste from contiguous reservations. Continuous batching admits new work as requests finish, improving use of the GPU. Prefix caching reuses computation for shared prompt beginnings across requests, while KV cache quantization reduces the memory occupied by cached state. Saving memory can enable longer contexts or more concurrent requests; it does not by itself establish faster generation or better answers.

• Serving engines need workload-specific evaluation. Jain recommends vLLM as a starting point for standard serving workloads and considering SGLang when repeated prefixes and branching agent workflows make cache reuse especially valuable. The comparisons he and Sah developed remain specific to their tested setup. Their practical lesson is to measure the workload the service will actually run rather than treat one benchmark as a universal ranking.

Across AgentShip and his inference handbook, Jain works on the infrastructure that makes AI applications usable beyond a prototype. He gives engineers concrete ways to separate application logic from runtime responsibilities, calculate resource needs, and distinguish memory savings from latency and throughput gains.