# Trading Desks to Clinical Trials: Parallels in Applied Vertical AI Ayush Bhardwaj, Allos AI AI Engineer World's Fair 2026 [Talk page](https://ai.engineer/talks/Yphdry8ttAQ-trading-desks-clinical-trials-parallels-in-applied) · [Video](https://www.youtube.com/watch?v=Yphdry8ttAQ) ## Trading Desks to Clinical Trials: Parallels in Applied Vertical AI A useful vertical AI system needs more than a working agent: it needs a narrow job, proprietary evidence, and experts who can judge whether its outputs create value. ### Different industries, the same engineering problem [0:15](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=15s) What changes when an applied-AI engineer leaves a hedge fund for a pharmaceutical startup? Ayush Bhardwaj made that move, taking responsibility for technology and applied AI at Allos. The setting changed, but the practical problem remained: build a system that can perform a specific industry job well enough to be useful. **Applied vertical AI** means AI built for one particular industry. Google Translate is a general-purpose application that can serve many industries; Allos uses AI for drug development. Legal-tech applications offer another example: the product is organized around a professional task, rather than a capability that happens to have many uses. ![Slide defining applied vertical AI as AI built for one industry, with Google Translate and Allos examples side by side.](https://ai.engineer/api/talk-essay-media/1fd5af29709d9c46dfa228aea5ae55e7ce7211749f6a2997e1257579f1b460a3) Applied vertical AI: Google Translate contrasted with Allos building drugs using a lab’s own data. Bhardwaj expected the move to change his work substantially. A hedge fund wants decisions fast and mostly right: individual losses can be acceptable if the overall strategy wins. Pharma puts much more weight on correctness. He illustrates that difference with a fifteen-year development horizon and a willingness to allow another week to get something right. Those are very different operating environments, yet the core applied-AI work felt remarkably similar. Conversations with practitioners in legal AI and property technology suggested that the development steps could be shared across industries. The first shared requirement is economic. Asking whether agents reach production sets a low bar—Bhardwaj jokes that even teenagers deploy them. The harder question is whether the system makes or saves more money than the complete investment required to build and operate it. He reports that agents delivered economic value at both of his employers, based on his own experience. **Deployment is a milestone; useful return on investment is the test.** ### Give the agent one narrow job [3:47](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=227s) The recipe begins with **problem formulation**. A request to find the top three investment opportunities leaves almost everything unspecified: the market, the industry, the comparison criteria, and what makes an opportunity attractive. Narrow it in sequence: 1. Choose a market: US equities. 2. Choose an industry: information technology. 3. Define the task: rank the stocks. 4. Specify the ranking criteria: capital expenditure or AI investment. The resulting job is small enough to describe and inspect. It also gives the next step—finding the right data—a concrete target. ![Step 01, “Formulate the problem,” with examples of flagging a mispriced bond and ranking molecules to test next.](https://ai.engineer/api/talk-essay-media/b5cb70c6c9f556cc8c6e824233df554c9636ee6409acb16608ffd68835e9783e) Formulate the problem: pick one narrow job for the agent. There is no requirement that one agent own the entire workflow. Bhardwaj’s joke is that there is no tax on building more agents: give each one a specific task instead of expecting a single agent to do everything. Pharma follows the same pattern. Break the process into steps, then ask pointed questions at each step. ### Find the evidence competitors cannot simply download [5:10](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=310s) Next, identify the data. News and sell-side reports from JPMorgan or Morgan Stanley are common financial inputs. In scientific work, arXiv, PubChem, and published research are widely available. Those sources can be useful without distinguishing your application from **ChatGPT** or **Claude**. The differentiating input is often proprietary evidence. That evidence may be expensive to buy, or its owners may refuse to sell it. Start by looking inside the organization. Bhardwaj imagines a company with three years of accumulated, unstructured records and suggests that an LLM workflow could structure them overnight. The practical opportunity is to turn existing work into usable records; the overnight estimate is an illustration, without a specified corpus size or validation requirement. | Industry | Valuable internal evidence | What it preserves | |---|---|---| | Finance | Trade theses | Which trades worked and why | | Pharma | Failed experiments | Outcomes harder to obtain from successful published results | These records contain the reasoning and negative evidence that a collection of public documents may miss. A failed experiment can matter precisely because it did not become a successful result worth publicizing. ### Model the practitioner’s work, then make it observable [6:24](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=384s) With a task and data in hand, write the prompt around how a practitioner would solve the problem. Bhardwaj initially frames this as modeling the person whose work the system might replace, then qualifies that he is not advocating replacing people. The useful engineering instruction is to **encode the human process as multiple steps**, rather than merely request a polished answer. For the US IT equities example, a prompt can make that decomposition explicit. This Python representation separates the task from the supplied records and gives the output a reviewable structure: ```python import json def ranking_prompt(records: list[dict[str, object]]) -> str: return "\n\n".join([ "Task: rank US information-technology equities " "by capital expenditure using the supplied records.", "1. Identify each company's reported capital expenditure " "and reporting period.", "2. Identify missing values or incompatible reporting periods " "before comparing companies.", "3. Rank comparable records by reported capital expenditure, " "highest first.", "4. Return the ranking, supporting record identifiers, " "and records excluded from comparison.", "Records:\n" + json.dumps(records, ensure_ascii=False), ]) ``` The comparison and output rules make the earlier narrow task concrete. Whether those rules capture a useful investment question still requires a practitioner’s judgment. Then add **observability**. Traces let the team see what the application did, understand its behavior, and debug it. Bhardwaj characterizes the first four steps—task definition, data, prompting, and observability—as the easy part, pointing to an agent implementation that fits on one screen. His image of a highly productive engineer building it in minutes makes a competitive point: the basic implementation is easy to reproduce. Proprietary data is his explicit exception. ### A working agent does not tell you whether it works [7:35](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=455s) Iteration seems like the obvious next step. At the hedge fund, however, Bhardwaj discovered that he could build the application without being able to tell whether its output was good. He was not a trader; later, in pharma, he was not a specialist in biology or chemistry. The missing capability was judgment, not implementation. Engineers already have a mental model for assessing generated code. They can recognize weak code or decide that a much-hyped model is less useful than advertised. That competence does not automatically transfer to a trade thesis or a specialist scientific task. A vertical AI project can therefore look finished to its builders while producing outputs its intended customers would reject—just as engineers reject an inferior coding assistant. Bhardwaj tried to solve this with **LLM-as-judge**. In his account, the attempt failed because fluent specialist language was not a reliable signal of useful specialist reasoning. A judge can reward jargon without recognizing whether an output creates financial alpha or other practical value. His qualification matters: the model needs some way to learn the domain’s standards. Asking another ungrounded model to evaluate the answer does not supply those standards by itself. ### Why verification and missing data become bottlenecks [9:36](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=576s) **Reinforcement learning with verifiable rewards** benefits from tasks with accessible checks. Math may have an answer key; code can be checked for compilation and subjected to further tests. The specialist tasks Bhardwaj describes do not offer an equally straightforward verification signal. A convincing trade thesis is not validated merely by being well formed. Errors can also propagate through subsequent reasoning steps, a concern he associates with Yann LeCun. The other bottleneck is evidence that never became public. For finance and pharma, Bhardwaj argues, the data needed to learn valuable professional judgments is often held by organizations that have strong reasons to protect it. ![Two-panel slide titled “Why you can't just iterate,” with headings “The model can't verify itself” and “The data was never there.”](https://ai.engineer/api/talk-essay-media/1913edebde40018769df676a3842921c6d690e014350e0c86975db47b33a82aa) Why you can’t just iterate: self-verification and missing proprietary data. His finance example is portfolio disclosure. Form 13F reporting applies to institutional investment managers exercising investment discretion over at least $100 million in covered securities, with quarterly filings generally due within 45 days after quarter-end. Bhardwaj points to a chart connecting disclosure with lower hedge-fund returns: competitors can inspect holdings and reverse-engineer a strategy. The useful mechanism here is the incentive to protect an informational advantage; he does not supply a numerical return decrease in the spoken explanation. The pharma example needs a narrower legal scope than the talk’s description of every trial. [Federal reporting obligations](https://www.fda.gov/science-research/clinical-trials-and-human-subject-protection/fdas-role-clinicaltrialsgov-information) apply to certain applicable clinical trials, not every trial or failed laboratory experiment. The [FDA’s 2026 reminder](https://content.govdelivery.com/accounts/USFDA/bulletins/412bbc8) found no submitted results for 29.6% of studies highly likely to require reporting—not 29.6% of firms. It reports reminders to more than 2,200 companies and researchers associated with more than 3,000 trials; some recipients may instead have needed to complete quality-control review. Missing public evidence is the relevant problem, but those figures do not establish that every omission concealed an unfavorable result. Bhardwaj compares valuable private evidence to a chicken that lays golden eggs: selling it can mean giving away the source of future advantage. He argues that frontier-model providers such as OpenAI and Anthropic consequently lack crucial protected data. Nor, in his hypothetical example, can a lab simply hire a trader for $100 an hour to annotate it: confidentiality agreements and the economics of specialist labor obstruct that route. This is his explanation for why public-model capability alone does not eliminate the need for internal expertise. ### Put the intended user inside the development loop [11:41](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=701s) The practical response is to **hire the person you want to sell to**. At the hedge fund, this relationship already existed: the trader was Bhardwaj’s boss and collaborator. At Allos, the engineering team had to bring in a scientist. Bhardwaj reports that the hire changed the direction of the tools and improved their reception with large pharmaceutical companies. The outputs began to speak the customer’s language instead of generic LLM jargon. The expert’s role starts before final evaluation. They can sharpen an overly broad query and help curate the evidence used to answer it. Engineers know which research sources and practitioners deserve attention; traders and scientists bring the equivalent discrimination to their own fields. Source selection is therefore part of the learning loop, alongside prompt refinement. The expert also supplies the order of operations. A problem with five steps cannot necessarily be solved by performing those steps in any order. Decomposition, gradual refinement, and final judgment need a logical flow. Repeatedly translating that flow into the application is how lived professional experience becomes agent behavior. The expert is not merely an approver at the end of an otherwise fixed pipeline. ### Choose how expert feedback changes the system [14:10](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=850s) Expert feedback can enter the system through several mechanisms: | Method | Learning signal | How it changes behavior | |---|---|---| | Supervised fine-tuning | Human demonstrations | Trains the model to imitate examples | | RLHF | Human preferences | Trains a reward model used for reinforcement learning | | Rubrics as rewards | Human-written criteria | Uses AI grading to guide improvement | Bhardwaj describes the rubric approach as reinforcement learning from AI feedback. It can make expert criteria reusable, but self-grading against a rubric can also create an echo chamber. The criteria and the grader still need scrutiny. His preferred starting point is **error analysis**: inspect the logs collected through observability, identify where the model went wrong, and correct the problem. This does not require changing model weights. He calls it the cheapest and highest-return starting point, a practitioner recommendation rather than a measured comparison in the talk. Once the team understands which errors remain, it can decide whether more expensive adaptation is warranted. He presents RLHF as an eventual gold standard in his industry, not a prerequisite for the first useful improvement. Model adaptation also creates recurring work. After a team fine-tunes a base model, a new release from a provider such as Alibaba Cloud or DeepSeek can make another round of fine-tuning attractive or necessary. The cost is therefore not just the first training run; it includes maintaining the adaptation as the underlying models change. ### Grow the learning loop, then sell the useful result [15:54](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=954s) As more practitioners join the loop, they bring more queries and expose additional tasks. Scope and data grow together. The development process itself now produces a proprietary dataset: examples of what works, what fails, and how experts distinguish the two. When the application becomes useful enough, release it to external paying users. The learning loop continues rather than ending at launch. Bhardwaj returns here to the difference between deployment and value. He invokes a high enterprise-agent non-production percentage attributed to Stanford’s AI Index, but the exact percentage and attribution are not established well enough to use as a statistic. His substantive objection stands independently: a system may be easy to deploy and still fail to work well enough to justify its cost. In his experience, finance and pharma demand economic usefulness quickly. A promise that the system may improve in two years, or become cheaper in three, is not enough. The development sequence matters: formulate the task, source the data, model the work in prompts, observe execution, bring in the user, and sustain an expert-led learning loop. Ship when that process produces meaningful value beyond what customers can already get from Claude or ChatGPT. ### AI is in the expert’s loop [17:45](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1065s) This deployment model puts responsibility with the expert. Bhardwaj distinguishes **AI in the loop**, or `AITL`, from **human in the loop**, or `HITL`. The latter suggests an AI-led workflow that a person checks. In the finance and pharma applications he describes, the expert owns the work and AI assists. For a trader, the assistant might generate five candidate trade theses. Those are proposals, not executed trades: the trader decides which, if any, might work in the market. In pharma, AI can assist with drug candidates, but the expert still chooses which to pursue. The value is reduced expert effort while preserving professional control over the consequential decision. Bhardwaj does not quantify the time saved. He expects that arrangement to persist because he sees causal reasoning as a boundary for autonomous decisions. Correlation with past examples is not, in his view, sufficient to decide what will happen next. Invoking LeCun’s distinction between text-based statistical learning and models of the real world, he argues that the system needs to reason about causes. This is Bhardwaj’s proposed threshold, not an established test of autonomy. ![Slide titled “Human-in-the-loop? Not yet.” contrasts human checking AI work with AI assisting an expert, followed by text about cause rather than correlation.](https://ai.engineer/api/talk-essay-media/c5e0e3b9546b9d279ad7eb38b40dc9e20f50d6d7f6a238e972bd8992fc67992f) Human in the loop versus AI in the loop: the expert does the work, and AI assists. He calls that threshold an AGI line. Beyond it, he imagines people creating drugs or market strategies through something resembling vibe coding. Those are speculative future possibilities; the working product described here remains an assistant to an expert. **The durable advantage is the expertise and evidence accumulated through that work.** Models, infrastructure, and ecosystem subscriptions are available to competitors too. The harder asset to reproduce is a body of domain judgment joined to internal records and other valuable data that is not on the public internet. No subscription supplies that complete advantage. The team has to curate it. ## Resources - [FDA's 2026 clinical-trial reporting reminder](https://content.govdelivery.com/accounts/USFDA/bulletins/412bbc8) — FDA announcement explaining missing trial results, reporting exclusions, and reminders sent to more than 2,200 companies and researchers. - [Stanford's 2026 AI Index](https://hai.stanford.edu/ai-index/2026-ai-index-report) — Official overview and chapter links covering AI capabilities, adoption, economics, science, and medicine. ## Slides ### Applied Vertical AI Google Translate serves anyone translating between languages. Allos builds drugs with AI using a lab's own data, illustrating AI tailored to one industry's work. [1:07](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=67s) [Defines applied vertical AI and contrasts Google Translate with Allos’s drug development work.](https://assets.ai.engineer/sites/aiecode2025/sha256/d1334d7f5e352e519a778a716a16e3da956c9216236974052bc82a6cf1c507da.webp) Applied Vertical AI Applied AI built for one industry. It runs on that field's own data and does the work people there actually do. APPLIED AI Google Translate translation between languages, for anyone APPLIED VERTICAL AI Allos building drugs with AI, on a lab's own data 02 ### A different time to market Moving from a hedge fund to pharma-tech meant a shift from roughly 1 month to launch a new strategy to 10 years for a new drug. The speaker expected a different world, with pharma placing greater emphasis on getting things right. [1:58](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=118s) [Compares one month for a new strategy with ten years for a new drug after a move from a hedge fund to a pharma-tech startup.](https://assets.ai.engineer/sites/aiecode2025/sha256/dc948b83a1e66a7ae0d76e57d1bd55532641fb4da0c41b8a450831536f75ec1c.webp) I left a hedge fund for a pharma-tech startup. TIME TO MARKET 1 month > 10 years a new strategy a new drug I was entering a completely different world. 03 ### Nothing changed Moving from a hedge fund to pharma expanded the speaker’s job, but the core applied AI work stayed the same. Conversations across legal and real estate tech revealed similar building steps. [2:30](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=150s) [States that the DNA of building is similar whether time to market is weeks or years.](https://assets.ai.engineer/sites/aiecode2025/sha256/d0665cf3e017c3ec786bb501bfd4ebad40a685966729f3f20c5880cf182db8cc.webp) Nothing changed. The DNA of building it is similar, whether your time to market is weeks or years. 04 ### Does anyone actually ship this? The slide cites 88% of organizations using AI and 50%+ of US businesses paying for AI products. The speaker asks whether deployed agents earn or save enough to justify their cost. [3:10](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=190s) [Reports that 88% of organizations use AI and over 50% of US businesses pay for AI products, then asks whether agents work.](https://assets.ai.engineer/sites/aiecode2025/sha256/e68c92a47bd0ab5336dc10f9fd562754872070658b1f45de0773e63101a6ec7a.webp) “But does anyone actually ship this?” 88% of organizations now use AI Stanford AI Index 2026 50%+ of US businesses now pay for AI products Ramp AI Index, Apr 2026 People ship agents everywhere. The only question is whether they work. 05 ### Formulate the problem Give each agent one narrow job: flag a bond that looks mispriced or rank molecules to test next. The speaker recommends breaking larger processes into specific tasks. [3:46](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=226s) [The first recipe step is to pick one narrow job, illustrated by bond mispricing and ranking molecules to test.](https://assets.ai.engineer/sites/aiecode2025/sha256/04c8e3e7cac7e60a466c9fadef81b86ecca34841733634f06cb6a9a76fba25c9.webp) THE RECIPE 01 Formulate the problem What is the agent actually for? Pick one narrow job. IN FINANCE flag when a bond looks mispriced IN PHARMA rank which molecules to test next 06 ### Identify the data The speaker argues that proprietary data can distinguish an application from general models, citing finance trades and pharma’s failed experiments as examples. [5:10](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=310s) [The second recipe step highlights proprietary data, with examples from finance and pharma.](https://assets.ai.engineer/sites/aiecode2025/sha256/76f3a9c0478a3a53feaa12ef91a88ec136f1f985b3737f307c934a3c50ca9862.webp) THE RECIPE 02 Identify the data Proprietary data is the moat. The base model never saw it. IN FINANCE trades, positions, alternative data IN PHARMA assays, failed experiments, trial data 07 ### Write the prompts Encode the steps a person would follow into prompts: a trader’s checklist for a thesis or a scientist’s protocol for a screen. [6:22](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=382s) [Recipe step 3: encode how work gets done, using a trader’s checklist or a scientist’s protocol.](https://assets.ai.engineer/sites/aiecode2025/sha256/d82f6a7aa0a263504ed74e8e17d1d4d8b59235bd7de22162123acf46ae787fbb.webp) THE RECIPE 03 Write the prompts Encode how the work actually gets done. IN FINANCE the trader's checklist for a thesis IN PHARMA the scientist's protocol for a screen 08 ### Observability Observability helps teams understand and debug agents: finance logs tool calls and numbers pulled, while pharma traces each step and citation. [6:50](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=410s) [Recipe step 4: observe agent activity by logging finance tool calls and tracing pharma steps and citations.](https://assets.ai.engineer/sites/aiecode2025/sha256/f0f3d485e90749a6e558318a0a3c13e91950776ccd0ec4568b2393d30c506016.webp) THE RECIPE 04 Observability Can you even see what the agent did? IN FINANCE log every tool call and number it pulled IN PHARMA trace every step and citation 09 ### The easy part fits on one screen A compact agent combines tools, a prompt and a run loop. The speaker says this setup is easy to reproduce, with proprietary data as an exception. [7:11](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=431s) [Example agent code uses tools and a prompt; a callout says the moat is not in the code file.](https://assets.ai.engineer/sites/aiecode2025/sha256/c26a089e4204f9e0e36dd33694f553f70cad0b4a75227c7794cd849f7e744890.webp) The easy part fits on one screen # Steps 1-4: a mythical 10x engineer ships this in minutes agent = Agent( tools=[market_data, research, backtest], prompt=load("trader_v7.md"), # the real IP ) answer = agent.run(todays_question) Where's the moat? Not in this file. Tools, a prompt, a run loop. Every team here can write this. 10 ### Iterate? After the first four steps, the speaker turns to iteration and recalls wondering how hard it could be when joining a hedge fund. [7:41](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=461s) [Recipe step 5 asks: Iterate? How hard can it be?](https://assets.ai.engineer/sites/aiecode2025/sha256/cbc3a940e094cb9e8543397185c2185e453cf0a093fc06ff795665e036102842.webp) THE RECIPE 05 Iterate? ...how hard can it be? 11 ### I could build it, but couldn't tell if it worked The speaker could build an AI agent but lacked the trading expertise to judge its output. That makes iteration difficult even when the software appears complete. [7:55](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=475s) [The speaker says they could build the agent but could not tell whether it worked, describing evaluation as a barrier for vertical AI projects.](https://assets.ai.engineer/sites/aiecode2025/sha256/db5d25e727d8fb8a0ebc4bda344f0f4283d2a5373ae16a974eaa47d2b86a192b.webp) I could build it. I just couldn't tell if it worked. Steps 1-4 are the easy part. Then you hit the wall: the part a 10x engineer can't speed-run, where most vertical-AI projects quietly die. 12 ### You can't automate the judge The speaker argues that an LLM judge can mistake jargon for alpha. The pseudocode replaces its judgment with a trader’s review, relying on someone who has lived the problem. [9:11](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=551s) [Evaluation pseudocode favors a trader’s review; a callout argues that an LLM judge cannot distinguish alpha from jargon.](https://assets.ai.engineer/sites/aiecode2025/sha256/f1de878cb7319855bcf1bff9b0550b44d4542ed3be711e97c81d059e4f80fac8.webp) You can't automate the judge # Step 5: did it actually produce alpha? def evaluate(output): # an LLM judge grades its own homework # return llm_judge(output) X return trader.review(output) # the human knows An LLM judge can't tell alpha from jargon. A human who has lived the problem can tell instantly. 13 ### Why you can't just iterate The speaker argues that trading and pharma lack the verifiers used in math and code, so errors can compound. Profitable trade theses and failed experiments also remain proprietary. [9:41](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=581s) [Two panels describe limits of model verification and unavailable proprietary data.](https://assets.ai.engineer/sites/aiecode2025/sha256/5f5e50c62388a28d677d2f01dad074a78412efc4295a51352b20ead7cfea292b.webp) Why you can't just iterate The model can't verify itself. RLVR made AI really good at math and code: they ship with an answer key, compilers and proofs. Trading and pharma have none. With no verifier the model can't tell a win from a loss, so its mistakes compound with every step (LeCun). The data was never there. The trade theses that make money and the experiments that failed are proprietary. Never for sale, never online. The exact data the model needs to improve is the data it never saw. 14 ### They don't publish the playbook The slide claims fund returns fall by 2.7%/yr after holdings are disclosed, while 30% of trials required to report results never do. It argues that valuable judgment data is missing from models. [10:25](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=625s) [The slide claims fund disclosure reduces returns by 2.7% per year and 30% of required trial results are not reported.](https://assets.ai.engineer/sites/aiecode2025/sha256/0142a60445d575ddf329158952de2cf527121e5a1c34d8e098c58b5e0579d046.webp) They don't publish the playbook. FINANCE HIDES THE WINS -2.7%/yr Make a fund reveal its holdings, and its returns fall. Secrecy is the edge. Journal of Financial Economics, 2017 PHARMA HIDES THE LOSSES 30% Nearly a third of trials that must report results never do. They just stay hidden. FDA, 2026 The work that encodes real judgment was never on the internet, so it was never in the model. 15 ### Hire the user Hire a domain expert to curate data and judge model output. The speaker says hiring a scientist helped the pharma startup build tools that spoke its customers’ language. [11:52](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=712s) [A slide recommends hiring a domain expert to curate data and judge model output.](https://assets.ai.engineer/sites/aiecode2025/sha256/fffeec4f7db399574e7721ecb4441e4060a599e1d62092def5a591d6bc277c59.webp) Hire the user. A domain expert who can curate data: the trader, the scientist, the one who knows when the model is capping. ### The expert builds the learning loop A domain expert breaks a workflow into agents, refines prompts and steps, and judges whether the output is useful or jargon. Their experience also guides data curation and the order of steps. [12:46](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=766s) [A domain expert contributes prompts, data curation and thinking models through three steps: decompose, refine and judge.](https://assets.ai.engineer/sites/aiecode2025/sha256/056cd5202f953e73992b337fc43eb7a763996bb1e760c522b5d8f061c8511d44.webp) The expert builds the learning loop domain expert = prompts + data curation + thinking models the user who's spent a career in the industry 1 Decompose Break their own workflow into agents, exactly how they'd solve it by hand. 2 Refine Push prompts, skills, and steps until the agent returns the first real “alpha,” not jargon. 3 Judge Know on sight when the output is jargon, not alpha, like a senior dev reading AI code. 17 ### Turning expert judgment into a signal Experts supply demonstrations, rankings and grading criteria, and inspect runs for errors. The speaker favors starting with error analysis, which can identify fixes without changing model weights. [13:59](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=839s) [Four methods turn expert judgment into signals: supervised fine-tuning, human feedback, reward rubrics and error analysis.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a9f6c2cb2c21d43d2dc297b17448f25b5dc8097b6c531a4d8f1ea84b0f862c4.webp) Turning expert judgment into a signal Supervised fine-tuning Retrain the model on tasks the expert has already done by hand. Reinforcement learning from human feedback The expert ranks answers, the model is trained to prefer the better ones. (DPO does this directly.) Rubrics as rewards The expert writes the grading criteria, that becomes the reward Error analysis The expert reads real runs and labels what went wrong, each pass sharpens the next. Each one turns scarce expert judgment into a reusable signal. 18 ### Ship it. Then mind the gap. More users generate queries and data that sharpen agents. The slide claims 89% of enterprise AI agents never reach production; the speaker argues the real test is whether they work and justify their cost. [15:50](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=950s) [A flow links more users and queries to more data and sharper agents. The slide claims 89% of enterprise AI agents never reach production.](https://assets.ai.engineer/sites/aiecode2025/sha256/2ebbfd279dc64224ba8c5cda0e14b27c72c89051cb6ca62001855201221f6589.webp) Ship it. Then mind the gap. More users > More queries > More scope > More data > Sharper agents the loop never stops, and every new user widens the moat 89% of enterprise AI agents never reach production Stanford AI Index, 2026 The ones that ship kept a domain expert in the loop. 19 ### Human-in-the-loop? Not yet. The speaker says finance and pharma still put experts in charge, with AI assisting to save time. The slide argues this will persist until AI can reason from causes, not just correlations. [17:08](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1028s) [Slide contrasts AI work checked by a human with expert work assisted by AI, and argues that causal reasoning remains a limitation.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c0fde3f53b2a58636763c8ffb24ac1fa3022bbdabb47c7b75c83104fc8a3605.webp) Human-in-the-loop? Not yet. MOST AI TODAY Human in the loop (HITL) The AI does the work. A human checks it. FINANCE & PHARMA AI in the loop (AITL) The expert does the work. The AI assists. It stays this way until AI can reason from cause, not just correlation: a real world model, not text statistics. Today's models pattern-match the past; they can't reason about a future that hasn't happened. 21 ### Domain expertise as a moat The speaker calls models, infrastructure and ecosystems commodities. Domain expertise and curated data, including data unavailable online, form the moat. [19:02](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1142s) [Slide describes models, infrastructure and ecosystems as commodities, and domain expertise as a moat.](https://assets.ai.engineer/sites/aiecode2025/sha256/318d5ccd814d61654826e38819b51b84d35be7fd34692c3a7ae5be0b0975fc6b.webp) model / infra / ecosystem = commodity domain expertise = moat ### AI Engineer World's Fair [19:53](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1193s) [AI Engineer World's Fair logo on a black background, with the event website below.](https://assets.ai.engineer/sites/aiecode2025/sha256/cf5bdef6819a65420305badbccd124642b3d38b356ffda21ea6d553a3b9b617e.webp) AI Engineer World’s Fair HTTPS://AI.ENGINEER ## Full transcript [0:00](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=0s) [upbeat music] Hello everyone. [clapping] [0:15](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=15s) So my name is Ayush Bhardwaj, and I did applied AI for a hedge fund, and now I do everything tech plus applied AI for a pharma tech startup, 'cause you know the way startups are. [0:26](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=26s) You have to do everything. We have multiple hats. So before I start the session, I would like to do a small survey. Can I get a raise of hands for all the engineers in the room? [0:36](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=36s) Oh, okay. That's a tough room. Now can I get a raise of hands for managers? [0:41](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=41s) Okay, just to be clear, managing AI agent does not count. [laughing] You have to manage people. Oh, okay, we have a few managers as well. Interesting. So it'll help me like fine-tune my talk a bit. [0:51](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=51s) So today my aim is to take you through the journey of how do you actually build and iterate in applied vertical AI, and my experience is from the hedge fund and the pharma tech company. [1:02](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=62s) So before delving deep into the recipe, I'll just like take you through what do I even mean by applied vertical AI 'cause I don't know, it, it sounds like a very weird term. [1:11](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=71s) It's like the vertical word is kind of forced. Uh, I won't lie, it is. I coined this term probably. So applied AI is like built for-- So vert-- Applied vertical AI essentially applied AI, but built for one very specific industry. [1:24](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=84s) Its, its aim is to simulate a job of a person in that particular industry, in a sense. So an example of applied AI is Google Translate, which is like general purpose, helps you translate. [1:35](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=95s) It could be used in education tech, and it can have like tons and various sorts of uses. Whereas Allos, which is my employer, the pharma tech company, we specifically build drugs with AI, so that's a very specific use case. [1:47](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=107s) Another examples of applied vertical AI field could be the legal tech firms that are now coming up with. You must-- I'm sure you must have heard about them. So those are like, uh, another good examples of applied vertical AI. [1:58](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=118s) So when I left the hedge fund, right, so I was expecting that the world will change for me 'cause, you know, hedge funds are like really fast and really pressure sensitive, whereas pharma is like, "Okay, we're gonna take fifteen years, but we're gonna do it right." [2:12](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=132s) Hedge fund was all about like, you need to do it fast and mostly right. It does not matter if we lose at one paradigm as long as we are overall winning. [2:20](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=140s) Whereas a pharma firm is like, "We have to be absolutely right. You can take a week more." And it was true. It's, it's a completely different world. But to your surprise, and to mine as well, nothing changed actually. [2:33](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=153s) My job increased, but the core part of my job, applied AI, remained the exact same, and I cannot, uh, express how surprised I was 'cause I thought that it'll be a complete different thing, but apparently it was not. [2:46](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=166s) So, uh, so then I spoke to other people as well across legal AI and the people coming up with the proptech firms, which essentially the real estate tech firms. [2:55](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=175s) And I realized that everyone is kind of building the applied vertical AI in a very similar way. I could see some steps that could be essentially abstracted out, and that's what we'll do today. [3:06](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=186s) So, uh, before, again, delving de-di-uh, deep into that, I received a few reach outs saying, "Are people actually putting agent into production?" And I was like, "This is such a wrong question to ask." [3:17](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=197s) Everyone is putting agent into production, even like fifteen year co-- uh, [REDACTED:age] kids these days. The question to ask is whether they actually work, whether they actually make or save money, whether they justify their ROI, whether, uh, they are making way more than the amount we are investing into it, like end to end. [3:34](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=214s) And I can say from my anecdotal experience, yes, at the both places I worked, the agent either saved the money or made more money. So with that, let's get started. [3:47](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=227s) So the recipe, I'll take you through a series of seven steps roughly, and try to like make this process as simple as possible. So the first step is formulate the problem. [3:58](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=238s) So this is sounds like very trivial, but a lot of people, specifically startups, get this wrong. They just try to do too much at once. Whereas from what I have learned and what I think a lot of colleagues would agree, you need to pick a very narrow task. [4:10](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=250s) You just cannot ask it to do everything. A good example for this could be, let's say if you build something in finance, you won't ask it to like, "Hey, can you fetch me top three market opportunities that I could invest in?" [4:21](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=261s) No, that won't work. You have to be like very specific. Like you pick a market, you say, "Let's take the US equities," then you pick a industry, let's take IT, and then you ask it to like rank stocks based on some parameters like capital expenditure or let's say the AI, um, uh, investments. [4:40](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=280s) So you pick like very specific things, and then you, uh, sort of formulate a very narrow job for the AI agent to do. And you can build like n number of AI agent. [4:49](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=289s) Last I checked, there was no tax on building more AI agents, so why do you want your single agent to do everything? So this is important, and this in the same, uh, in the pharma context, it's the exact same. [4:58](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=298s) We just break down the process into steps and then ask really pointed questions with the agent. We model our agent for a task. So once we have our pom-- problem right out of the way, we know what we're trying to solve. [5:10](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=310s) The next step is identify the data, and I cannot stress this enough. This is a really, really, really important step 'cause everyone has news data. Everyone has like sell-side reports from JPMorgan, Morgan Stanley. [5:24](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=324s) Uh, everyone has the arXiv preprint server or PubChem or your research papers, right? But what actually makes your application better than, let's say, ChatGPT or Claude? It is your proprietary data. [5:36](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=336s) So the thing with proprietary data is it's really expensive to buy, and most people won't sell it to you, so you need to curate it by yourself. Imagine your organization has been working for three years, right? [5:45](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=345s) They already have a lot of data. It's just unstructured. And in the age of LLMs, I think this is a very fairly easy task to make unstructured data into structured data. [5:55](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=355s) Like a LLM workflow could do it overnight. So to give you a great example of the- [6:02](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=362s) proprietary data that finance industry has, it's the trade thesises, which is like what trade work and why it worked. And in pharma, it is the data for failed experiments. [6:13](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=373s) 'Cause successful experiments data, yes, you can get it, but failed experiments, that's relatively hard to get. So now we have the problem, we have the data. What's the third step? [6:24](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=384s) That is to model the problem, like write the prompt. So while writing prompt, you-- like, what we should aim is to model it after the person who you are trying to replace. [6:35](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=395s) I mean, that's the hypothesis, but yeah, no offense, we are not trying to replace anyone with AI. But that's the ideology behind writing prompts, encode how a person would solve this job into multiple steps. [6:46](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=406s) So it's just like a, like a mental model. So this is again, fairly simple. Next thing, observability. I'm sure you have been to this conference at three years, and this word, I think, I don't know, you'll be hea- hearing about like a thousandth time. [6:58](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=418s) There are tons of observability provider. If you can see it, you can fix it, so you need observability to see the traces, understand what your, uh, AI application is doing, and debug it. [7:11](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=431s) So sorry, but all of this was the easy part, to be honest. All of this fits one screen. The mythical 10X engineers can do this stuff in minutes. Like literally, this is the code you precisely need to build an AI agent, so that's why it's not the moat. [7:27](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=447s) Uh, of course, except your proprietary data. So what do you do now? What do you do after doing the first four steps, [7:35](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=455s) which is observability and prompt and, like, uh, getting the data right and everything? You iterate. Now, the thing with iteration is, like, when I joined the hedge fund, I thought how hard it can be. [7:49](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=469s) I mean, everyone can iterate. I mean, we have been iterating our whole life for each of the task. But to be honest, I could build it, but I just could not tell if it worked, 'cause [8:02](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=482s) I'm not a trader. I'm not someone who has a PhD in biology or chemistry. I just don't understand what the model is saying, what is the output of my AI agent is. [8:12](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=492s) And since most of you are engineers, you would relate. You can instantly tell that Sonnet 5 sucks because you have your own training. You understand, okay, this code is not great code. [8:23](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=503s) Whereas some X model that's a fail five, you see, okay, this is great, but not as great as the hype is. 'Cause you have been trained for this for life. [8:30](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=510s) You have a mental model to judge these things, but you just do not have the same kind of mental model when it comes to, like, predicting trade thesises or doing, like, really specific tasks that vertical AI industry does. [8:44](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=524s) And this is also the place where, like, a lot of vertical AI projects quietly die because on the surface it looks like you have made it, you have built it, let's put this into production and start selling it. [8:55](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=535s) But no one would buy it, the same way you won't use an inferior coding model. So as an engineer, when I ran into this, I just couldn't accept, honestly. [9:04](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=544s) I thought, "No, there's certainly more that I can do. We don't need other people." So I thought I could, uh, LLM as a judge my way out of it. [laughs] [9:14](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=554s) And this was a really, really stupid mistake, to be honest, 'cause what LLM is essentially doing, it's, it's predicting the next probable word. So if you see, it's just like jargoning its way out. [9:24](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=564s) It does not understand what alpha means. It does not understand how to actually create value unless you have, like, taught it some way. And whereas a human can just tell it instantly what's bullshit and what's not. [9:36](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=576s) So I'll just try to delve a bit more deeper on why you can't just iterate. So first thing is that model cannot verify itself, specifically in these fields, because reinforcement learning via verifiable rewards is really good at math and code because you have, like, answer keys, you can verify your code is, uh, compiling or not, and there [9:56](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=596s) is tons of step. You can just model, uh, the complete thing around this. But when in these fields, there is just no way to model it. And, and let's say if any error gets in, it's just compounds with every stuff, and that's what LeCun seems to think as well. [10:11](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=611s) And now the more important part that we touched upon previously, the data. So the interesting thing with pharma and finance is the data was never there, and I'll explain to you why. [10:27](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=627s) So any institutional manager holding over hundred million dollars in qualifying US equities are forced to publicly file their holdings, long position holdings, every quarter. And once a hedge fund does this, this is the percentage decrease in their returns because everyone just sees those reverse engineers and takes away their moat. [10:48](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=648s) And when it comes to pharma, right? So this is the number of, uh... So by law, you are, like, required to di- disclose every clinical trial, pass or fail, you have done. [10:57](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=657s) But thirty percent of the firms, which is like nearly one-third of firms, never do. And in like 2026, FDA had to, like, publicly remind over, I don't know, but two thousand sponsors that they are, I mean, doing injustice by not, uh, releasing unfavorable results. [11:14](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=674s) Because this is the exact data which helps the model think, which helps your LLM actually reason through these complex and niche industries. And they hide it because for them it's like a chicken laying golden eggs. [11:26](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=686s) Why would they sell their chicken? So naturally, neither OpenAI nor Anthropic has that, has this data because it's like gate-kept. You just cannot hire a trader for hundred dollars an hour and have them annotate that stuff because there are, like, lots of NDAs, and they definitely earn more. [11:41](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=701s) So okay, now I have told you about tens of problems right? Now, you will naturally think, "Okay, yeah, right. Then what do we do? How do we build a startup in like a vertical spa- space?" [11:53](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=713s) So- Very self-explanatory. You hire the person who you want to sell it to, 'cause there is, to be honest, no other way around. I have tried a lot of stuff. [12:04](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=724s) You just need to hire the user. In finance, in a hedge fund, this was very easy because the user was kind of like my boss, the trader. We worked together. [12:13](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=733s) But in the pharma tech startup, it was very weird. We are, like, a bunch of young engineers, and we are like, "Oh, we need a [REDACTED:age] scientist in our company to tell us what to do? [12:22](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=742s) Yeah, I guess we do." And then we hired someone, right? And that someone actually changed the trajectory of our tools. Our tools started making sense when we pitched to the other pharma companies, the big ones, the big pharma. [12:34](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=754s) They started liking our tools because it kind of spoke their language versus the normal jargonese LLM language. So once you have hired the user, let's say, then what would you make that user do? [12:47](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=767s) You try to build a learning loop out of it. The domain expert can start at the, like, a very, very low level, the ground level, where they just think about prompts. [12:56](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=776s) Okay, uh, uh, I mean, let's not ask, uh, LLM to do this. Let's ask a very specific query again. They'll help you curate data. Just like engineers know which conference are bullshit, which are not, which, uh, research paper sites are great, which are not, which are, like, top leaders in engineering, which is-- which are just, like, influencers. [13:15](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=795s) Similarly, a pharma expert or, let's say, a trader knows which sources are more reliable than the other. So they help you curate their data, they help you, like, refine your prompts better, and they try to create, like, thinking models of how they would think about a problem. [13:29](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=809s) 'Cause, I mean, let's say if you s- if you, uh, follow five steps to solve a problem, right? You just cannot do it in, in any random order. There has to be a logical flow. [13:37](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=817s) There has to be a natural flow. That-- so that's what they, uh, try to curate, like decompose a problem, gradually refine, and then finally judge. So the person who sort of has lived through the complete of the industry that they're trying to revolutionize, their judgment is now, like, turning into agents. [13:56](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=836s) So that's what's happening behind the loop. So, uh, to do this, there are, like, again, multiple ways. I mean, each of these could have been a hour-long session on its own, and I wish I could take, but these are, like, few ways that I identified. [14:10](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=850s) Uh, I'll just, like, take, uh, you through them, like, really quickly in the interest of time. So supervised fine-tuning, I think most of you would know where, like, model mimics human de-demonstrations. [14:20](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=860s) Uh, reinforcement learning from human feedback is, like, a kind of a, a very efficient way where human preferences train a reward model. Then rubrics as a reward is, I, I like to call it reinforcement learning from AI feedback. [14:35](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=875s) This is because that you can-- human can just create a rubric, and then AI will just, like, grade itself based on that rubric and that improve its own processes. [14:44](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=884s) But again, there is a slight chance that you might run into an echo chamber with rubrics as rewards. And the cheapest of all, and I think the highest ROI, is the error analysis. [14:54](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=894s) Whereas the observability part that you set up earlier, you just analyze the logs plain and simple. You understand where model is going wrong, and then you just try to correct it. [15:03](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=903s) So this is where you have, like-- don't have to touch any weights and the most highest ROI way to get the impact from, like, start on. And once you understand, like, uh, what more you could do or if error analysis is solving or not, you can just gradually climb up the ladder and probably, uh, later on go [15:22](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=922s) to the ultimate reinforcement learning from human feedback. 'Cause that's, I think, in our industry, kind of the golden standard these days, that you need to do RLHF to actually get some edge. [15:33](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=933s) But, uh, certainly there are some pitfalls of it. Like for example, now there's GLM five point two, right? You fine-tuned it, right? [15:42](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=942s) Uh, Alibaba Cloud or let's say DeepSeek will release a newer model, then you have to fine-tune that too as well. So there is a cost. It's not cheap. So once you have done all this, you just create a loop, and you just, like, go onto that loop. [15:54](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=954s) You hired one user, you hire more users. They ask more queries, the scope increases, the data increases. At this point, you are kind of generating your own data. The exercise you have been doing in loop, right? [16:05](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=965s) That exercise itself is generating a very, I would say, a crazy data set of what works and what does not work, and this loop never stops. Once you feel confident enough in your application, you just ship it, provide it to the external paying users, and then you see the magic of it, that it actually works. [16:21](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=981s) So I just pulled this stat from Stanford AI Index report 'cause it's a really nice report that gives you an idea of what the state of AI is, and this says like eighty percent, eighty-nine percent of enterprise AI agents never reach production. [16:35](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=995s) Again, I disagree. Every AI reaches production, but it just fails to work or, like, justify its own cost. So that's the real thing. You can just build and ship AI agents whenever you want, but you need to justify ROI. [16:48](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1008s) And finance and pharma are two such industries where if it does not make money, it's shown the door. Simple. They won't, like, wait and say, "Okay, maybe it'll work in two years. [16:57](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1017s) Maybe the cost will be lower in by the third year." No, it has to instantly make money. It has to, like, hit the ground running, and if it does not, shown the door instantly. [17:06](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1026s) So just to summarize the seven steps that I feel are, like, good enough to give you an abstraction of how the vertical AI industry moves. You formulate the problem statement, you source your data sources, you prompt it well, you refine those prompts, you observe how your tool is performing. [17:24](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1044s) You don't iterate yet. You hire the user, and this user or users now play with the tool as much as possible. They, like, kind of form a learning loop, an endless learning loop that goes on. [17:35](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1055s) And at a point when you feel, yeah, it's, it's really delivering that alpha over, let's say, Claude and ChatGPT, you just ship it. You start earning money. So- [17:45](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1065s) One more interesting thing. So HITL is like kind of a thing. Everyone is like, "Yeah, let's add human in the loop." I would say not yet. Finance and pharma are still those two industries where it's AITL, AI in the loop, 'cause everything is, like, done by the expert, but the AI assistant really helps save time. [18:03](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1083s) Like for example, uh, it may take an X amount for a trader to form different trade thesises, and AI can just give him five candidate trade thesises. But which one would actually work in the market and which won't is the decision, uh, the decision still lies with the trader. [18:18](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1098s) And same for pharma. When you're, like, picking drug candidates, which one to pick, the expert still does it, but you just, like, reduce the time of expert by a lot, lot. [18:29](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1109s) So, and, and it will stay this way for really long. So, uh, for the models to actually make good decisions, they don't need to do correlation, they need to do causation. [18:39](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1119s) And as Ya- as Yann LeCun puts it, these are like text statistics, not real world models. You cannot just pattern match with past and use it to predict the future. [18:49](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1129s) And so we are, like, kind of not there yet. That's what I call as the AGI line. Once we are there, yeah, probably then models will just, like, make drugs. [18:57](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1137s) You will have vibe coded drugs. Someone would be vibe coding market. But yeah, not yet. So a final takeaway that I would call if, if, if there's one thing you are taking away from this talk, this is it. [19:10](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1150s) Model infra ecosystem, everyone selling you tons of stuff at this conference is just commodity. Everyone has it. If you have it, everyone has it. Everyone can pay X number of dollars for a subscription, but what is moat? [19:22](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1162s) And no one will come and sell it to you. You will have to curate it on your own. It's the domain expertise. You need your data, you need other people's data that is just not out there on the internet, and that, that's what will form your moat. [19:34](https://www.youtube.com/watch?v=Yphdry8ttAQ&t=1174s) So thank you for your time. I think you enjoyed the talk. And yeah, let me know if you have any questions. We can meet outside. Thank you. [audience applauding] [outro jingle]