POC to PROD: Hard Lessons from 200+ Enterprise GenAI Deployments

What does generative AI actually contribute when a customer needs an application built or an Oracle database moved to Postgres? Caylent works across those kinds of requests, serving startups through large enterprises. That breadth gives Randall Hunt a practical starting point: generative AI is one component of a solution, not a universal answer to whatever problem arrives next.

Building operations provide one example. BrainBox AI manages HVAC systems, and Caylent built an agent to support its work on building management and decarbonization. Hunt describes a North American footprint of tens of thousands of buildings; that is his account of the deployment scale. He also recalls recognition from TIME: ARIA appeared in its 200 Best Inventions of 2024. Water management and conservation provide another application. These are operational problems with a purpose beyond adding a conversational interface.

Nature Footage has a large stock-video collection: lions, tigers, bears, crocodiles, and other wildlife. Making it searchable requires more than putting one vector beside each file. The system also needs descriptions that capture what happens in a clip and when it happens. Nova Pro generates video descriptions, timestamps, and features, which the application stores in Elasticsearch for retrieval.

The visual retrieval path samples frames from a video, embeds those frames, and pools their embeddings into a representation of the clip. A text query can then retrieve visual content through a shared multimodal embedding space. Descriptions and vectors serve complementary purposes: one makes the video’s interpreted content searchable, while the other supports similarity across text and images. Hunt calls the embedding model “Titan V2 multimodal”; AWS documents its image/text model as Titan Multimodal Embeddings G1, distinct from Titan Text Embeddings V2. That naming distinction does not establish which model was deployed in this example.

A second video architecture processes both live sports and archival footage. Its pipeline combines conventional signal processing with model inference:

1. Extract the audio and generate a transcript.
2. Use FFmpeg to inspect audio amplitude and look for audience cheering as a simple highlight heuristic.
3. Generate embeddings from the text and video, identify behaviors with confidence estimates, and store the results.
4. Send event notifications through Amazon End User Messaging or SNS—for example, when the system identifies a three-pointer.

Cheering is a useful way to find candidate moments before asking a model to interpret them. It does not require the model to discover every interesting interval from raw footage alone.

The next improvement changes the model’s input. With a static camera angle, annotate the court’s three-point boundary with a prominent blue line. The question can then become: did the player cross the blue line? This makes the relevant boundary explicit instead of requiring the model to infer it from the unmodified scene. Hunt reports better results with this small augmentation, without giving a numerical benchmark. SAM 2 can assist with annotations; he presents it as an option, not as a verified component of this particular deployment. The architecture also names Postgres with pgvector and OpenSearch as vector-store choices.

Hunt’s delivery perspective spans physics work at NASA, database technology at 10gen before it became MongoDB, leadership of SpaceX’s CI/CD team, and customer engineering at AWS. He also recalls making a Transformer explainer in July 2017. At Caylent, that background feeds a customer-building practice covering chatbots, copilots, and agents. The useful distinction is what each application is supposed to accomplish.

Self-service productivity often starts with software an organization can buy. Some institutions still need a fine-tune or an application around it, but they also need to administer third-party tools, track API usage, and control sensitive data. Hunt describes on-premises networks and VPNs as places to inspect traffic for personally identifiable information and protected health information. He recommends SurePath, which Caylent uses, and mentions integration with Zscaler.

Business-process automation has a different target: recover time or money across an entire workflow. For a logistics customer handling receipts and bills of lading, Caylent puts a custom classifier before the generative models. Hunt reports faster and better results than that customer’s human annotators, but supplies no measured comparison. The architectural point is the sequence: classify the document before sending it into generative processing.

Monetization adds a paid capability to an existing product, often as a new SKU. An existing SaaS platform already has users and a workflow in which AI might create value. A generic chatbot, however, gives those users little reason to prefer one vendor’s implementation over another. The feature needs differentiation beyond the presence of a model.

The system’s inputs and expected outputs are its fundamental contract. Start there, then build evaluations that distinguish repeatable behavior from a successful response to one unusually favorable prompt. A demonstration shows that something can work; an evaluation set begins to establish whether it keeps working as the inputs change.

The hierarchy then moves through system architecture to particular models and tools. Those implementation choices should be expected to evolve. The specification can also expand as new output modalities become practical, but the business questions remain: why build this system, what result should it produce, and what return justifies the investment? That keeps model selection subordinate to the product’s purpose.

In the AWS stack Hunt presents, Bedrock and SageMaker form the service layer. He describes SageMaker as carrying a compute premium; running on EKS or EC2 is another option. Below that sit accelerator choices. Hunt cites approximately 60% better price-performance for Trainium and Inferentia than NVIDIA GPUs, without specifying workloads, chip generations, or a GPU baseline. He pairs that claim with a warning about less HBM capacity than an H200, likewise without identifying the AWS accelerator generation. Treat these as prompts for a workload-specific comparison, not portable benchmark results.

Hunt also points to newly announced P4 and P5 price cuts, recalling a maximum of 40%. The June 2025 AWS announcement actually advertised up to 45%, with that maximum applying to P5 three-year EC2 Instance Savings Plans. These are historical purchasing terms, not current prices. Trainium and Inferentia also introduce a programming requirement: the Neuron SDK. Hunt compares the Neuron Kernel Interface with XLA and TPU programming. Above the compute layer, model choices include Claude, Nova, Llama, DeepSeek, and other deployable open-source models.

Vector storage carries its own economic tradeoffs. Hunt prefers Postgres, while also pointing to MemoryDB for Redis-compatible persistence and vector search on AWS.

The central issue is where the index lives and how much memory it needs. Hunt warns that IVFFlat-style index planning can consume substantial RAM, then points to disk-backed storage and HNSW options in Postgres and OpenSearch. HNSW is not a promise that memory requirements disappear.

Prompt versioning and management are common infrastructure. Context management is a stronger source of differentiation: an application may know which page a user is viewing, what they have browsed, and other information relevant to the current task. Supplying that context lets the model make an inference grounded in the user’s situation rather than answering the same generic question for everyone.

Evals and embeddings still do not define a complete product. The design must reflect how people actually access information. A similarity query alone does not provide faceted navigation or structured filters. Postgres and OpenSearch are useful here because retrieval can combine semantic matching with the other constraints users need to express. Start with the access pattern, then choose the retrieval machinery that supports it.

Latency affects adoption alongside price. Caching can reduce actual waiting, while interface feedback can make unavoidable waiting easier to tolerate. Hunt argues that a slower, more expensive system is difficult to justify, whereas a slower but cheaper system may still succeed if its interface handles the delay well. The user’s experience includes the wait, not just the final answer.

Some delays are self-inflicted. A tool whose only job is to return the current date asks the model to request information the application already knows how to obtain. Supply it directly in the prompt instead. For example, application-side Python can construct the context without a model tool call:

from datetime import datetime, timezone

instructions = "Answer the user's question using the supplied context."
current_date = datetime.now(timezone.utc).date().isoformat()
prompt = f"{instructions}\n\nCurrent date (UTC): {current_date}"

Placement matters too. Frequently changing information near the beginning of a prompt can reduce prefix-cache reuse. Keeping stable instructions first and placing dynamic context afterward can preserve more reusable prompt content.

As models improved, prompt engineering proved more effective for Caylent than Hunt had expected, changing his earlier preference for fine-tuning. Moving identical prompts from Claude 3.5 to Claude 3.7 caused regressions in some customer cases and evaluations. Hunt reports zero regressions in Caylent’s observed move from Claude 3.7 to Claude 4. He also describes faster, better, cheaper inference in virtually every use case they encountered, without supplying benchmark conditions. That is an account of their upgrade experience, not a guarantee for another application or a future release.

Less prompt rework on future upgrades is a hope, not an established property of model releases. Each application still has to justify its inference economics. A capable Opus model may be inappropriate if its cost makes the feature unsustainable at the company’s expected usage.

Evaluation does not have to begin with an elaborate scoring framework. Using a slide he attributes to Anthropic, Hunt describes an incremental procedure:

1. Preserve the first manual test—the initial qualitative check—as an evaluation case.
2. Change the input data to create additional cases.
3. Define what success means. A Boolean result may be easier to judge than an artificial numerical score.
4. Run the cases, inspect the outputs, adjust the prompt, and repeat.

The useful transition is from an impression of one answer to a repeatable test of behavior. A small initial set gives the team something concrete to run as the system changes.

Continue iterating across output evaluation, prompt management, and the interface. The slide extends the process through A/B evaluation and feedback loops. Production readiness emerges from those repeated checks, while UX orchestration determines whether the system’s behavior feels useful to the person waiting for it.

CloudZero began with a chatbot for asking questions about AWS infrastructure costs. The application then moved toward generative UI: create a React component just in time and insert it into the response rendering so the answer can include an appropriate chart or other presentation. Generated components can be cached, and later prompts can describe available components for possible reuse. Hunt presents this as a way for the interface to evolve and personalize, made practical by sufficiently fast inference.

For users in remote areas, the limiting factor was connectivity rather than the sophistication of the answer. They received useful summaries of manuals, then faced approximately 200 MB PDF downloads to see the supporting material. Caylent’s adaptation was to render the relevant PDF page on the server and send that page image with the text summary. The user could read the necessary instructions and inspect the relevant source page without transferring the full manual. This solves a different interface problem from CloudZero’s charts or Nature Footage’s similar-video search, even though all three use AI within the product.

A hospital deployment exposed another mismatch. Caylent initially built a voice bot for nurses, but the noisy environment interfered with transcription and picked up other people speaking. The nurses in that deployment preferred a conventional chat interface. Knowing the user means understanding the conditions in which they work, not merely choosing the newest input modality.

Ordinary computation should handle arithmetic. Asking an LLM to calculate spends inference on work a computer can perform directly. Prompt optimization also needs to account for output tokens: the volume of generated text affects cost and performance, so producing the needed answer efficiently is part of the engineering task.

Other cost controls include prompt caching, appropriate tool use, and batch processing. Hunt cites a 50% discount for Bedrock batch inference; the historical announcement limits that discount to selected supported models and regions, relative to on-demand inference. It applies to asynchronous work, rather than establishing a universal discount for every model or interactive request.

Finally, seek minimum viable context: the smallest context that still supports a correct inference. That requires knowing what the user is doing, injecting the information that matters, and removing material that does not. Context optimization is an ongoing process of deciding what the model needs to reason over—not simply making every prompt shorter.