# How BlackRock Builds Custom Knowledge Apps at Scale

Vaibhav Page · Infant Vasanth

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/08mH36_NVos-blackrock-builds-custom-knowledge-apps-at-scale) · [Video](https://www.youtube.com/watch?v=08mH36_NVos)

## How BlackRock Builds Custom Knowledge Apps at Scale

BlackRock’s sandbox and app factory turn domain expertise into configurable extraction workflows, then package those workflows as applications that reach downstream systems.

Familiarity with LLM prompting, structured data extraction, and application deployment will help; no financial-instrument expertise is required.

### Turning information into operational work

[0:39](https://www.youtube.com/watch?v=08mH36_NVos&t=39s)

How do you build custom applications quickly when every step from incoming information to a completed trade requires specialized knowledge? At BlackRock, portfolio managers and analysts synthesize information, develop investment strategies, rebalance portfolios, and initiate trades. Investment operations supports that process: acquiring data, executing trades, handling compliance, and completing post-trade activities. Each domain needs internal tools, often with substantial complexity.

Infant Vasanth and Vaibhav Page organize these applications into four categories. Each offers opportunities to augment existing systems with LLMs:

| Application category | Work it supports |
| --- | --- |
| Document extraction | Extract entities from documents |
| Automation and workflows | Run multiple steps and integrate downstream |
| Knowledge search and Q&A | Answer questions through chat interfaces |
| Agentic systems | Coordinate actions using models and tools |

The shared challenge is getting these domain-specific applications into users’ hands quickly.

![Enterprise use cases slide with four columns showing document processing, extraction workflows, a knowledge graph, and agents with tools, each above a custom app.](https://ai.engineer/api/talk-essay-media/a7176dbe67194d018294ac636f377be3091a5e2ba0e088868aec0660bddead6a)

Four enterprise use cases: document extraction, automation and workflows, knowledge search and Q/A, and data agents.

### A prospectus must become a configured security

[2:19](https://www.youtube.com/watch?v=08mH36_NVos&t=139s)

A request from the new issue operations team, received three to four months before the talk, makes the problem concrete. When a company goes public or a stock splits, the team must set up the security in BlackRock’s internal systems before portfolio managers or traders can act on it. Reading the source document is only the beginning.

The simplified workflow has several handoffs:

1. Ingest a prospectus or term sheet into a processing pipeline.
2. Work with domain experts, including equity and ETF teams, to define what the complex instrument requires.
3. Produce structured output.
4. Have engineering implement transformation logic and integrate that output with downstream applications.

**Extraction alone does not deliver an operational application.** The output still has to become something the receiving systems can use.

Introducing model providers and trying new processing strategies adds iteration to an already lengthy process. Vasanth says the team tried agentic systems, but they did not work adequately for this use case at the time: too much instrument-specific knowledge remained in the experts’ heads. That experience motivates a framework that makes expert participation easier rather than assuming it can be removed.

### Prompts and extraction strategies evolve together

[3:56](https://www.youtube.com/watch?v=08mH36_NVos&t=236s)

The first bottleneck is expressing domain knowledge precisely enough for extraction. In one simple case, a prompt began as a couple of sentences and expanded to three paragraphs as the team described the financial instrument. Once prompts become substantial specifications, operators need to iterate, version them, compare alternatives, and evaluate performance against a dataset. Prompt editing without those surrounding capabilities leaves the team unable to tell whether a change helped.

The second bottleneck is choosing how to present the task and its evidence to the model. The options discussed include **retrieval-augmented generation**, chain-of-thought approaches, and direct in-context extraction. A vanilla corporate bond with a short source document may be simple enough to pass directly to a model. Documents thousands of pages long—and, in Vasanth’s examples, as long as ten thousand pages—create a different context-management problem.

Vasanth raises the question of sending more than a million tokens to OpenAI models. That is a model-specific capacity concern, not a universal current limit: [**GPT-4.1**](https://openai.com/index/gpt-4-1/) had already been announced with up to one million context tokens before the talk. Capacity alone does not establish extraction reliability. The practical task remains to vary prompts and strategies together while accounting for context limits, model capabilities, and provider differences.

### Deployment choices belong in the workflow

[5:52](https://www.youtube.com/watch?v=08mH36_NVos&t=352s)

Experimentation can stretch into months, after which deployment introduces another set of decisions. The application still needs distribution, access control, and a way to reach its users. AI workloads also require a choice of execution environment.

Vasanth illustrates the compute decision with an equity team asking to analyze 500 research reports overnight; he proposes a GPU inference cluster for that request. This is a workload example, not a measured throughput result. For new issue setup, he instead describes using a burstable cluster.

These choices should be defined well enough that application deployment approaches a CI/CD process. Cost controls belong alongside the deployment configuration: a workflow that is easy to prototype can still be expensive to operate. The GPU and burstable-cluster examples illustrate workload-dependent choices, rather than requirements for every extraction application.

### Separate expert iteration from application production

[7:00](https://www.youtube.com/watch?v=08mH36_NVos&t=420s)

Vasanth reports that the framework reduced development time for a complex application from three to eight months to a couple of days. The talk does not specify measurement boundaries or a controlled comparison, so this is the team’s reported experience rather than a benchmark. The architectural change centers on two components: a **sandbox** and an **app factory**.

Shared platform components handle the surrounding data lifecycle. The data platform ingests data, and an orchestration layer transforms it into forms that can be distributed through applications or reports. Above those foundations, the framework exposes the pieces that repeatedly slow application development: extraction templates, LLM strategies, extraction plans, transformers, and executors.

**The sandbox puts those modular pieces in the domain experts’ hands.** Experts can iterate on their extraction logic and configuration, then pass the resulting definition to the app factory. Vasanth describes the factory as a cloud-native operator that accepts a definition and produces an application. The division gives experts a place to refine the work while giving deployment a repeatable input.

![Architecture diagram showing Data Platform, Sandbox, App Factory, and Developer Platform, with a DocuFlow API and an orchestration layer containing four workflow types.](https://ai.engineer/api/talk-essay-media/d458b76f7d709838884183811f66875c19ebe1d415d04b373a7291bb4b1d0205)

DocuFlow architecture connects the sandbox and app factory with shared platform components.

### Encode dependencies, not just prompts

[8:33](https://www.youtube.com/watch?v=08mH36_NVos&t=513s)

Page begins the demonstration with a reduced version of the internal tool. The sandbox lets operators build and refine extraction templates, run them against documents, and compare the results. Familiar prompt-template features provide a starting point: fields have prompts, expected output types, and associated metadata.

Operators also need QC checks, validations, constraints, and **inter-field dependencies**. A callable bond supplies the concrete example: if the security is callable, its call date and call price must have values. Independently extracting three correctly typed fields is insufficient if their combination violates that rule.

The example template contains issuer, callable, call price, and call date. When adding a field, an operator specifies its name, expected type, and whether its source is extracted or derived. A derived field can be populated by a downstream transformation instead of another model extraction. Required status, dependencies, and validations complete the configuration.

The callable-bond dependency can be expressed concretely with JSON Schema. This schema fragment captures the conditional validation rule; it is not the sandbox’s configuration format:

```json
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "issuer": { "type": "string" },
    "callable": { "type": "boolean" },
    "call_price": { "type": "number" },
    "call_date": { "type": "string", "minLength": 1 }
  },
  "required": ["callable"],
  "if": {
    "properties": { "callable": { "const": true } },
    "required": ["callable"]
  },
  "then": {
    "required": ["call_price", "call_date"]
  }
}
```

When `callable` is `true`, omitting either dependent field fails this rule. Validation detects the missing values; it does not invent them or establish that a supplied price or date is factually correct.

### Carry extraction results into downstream systems

[11:02](https://www.youtube.com/watch?v=08mH36_NVos&t=662s)

Document management supplies the other input to an extraction run. Documents arrive from the data platform, receive business-category tags and labels, and are embedded. The UI exposes these capabilities alongside the framework’s modular components so domain experts can assemble their applications.

After a demo interruption, Page continues the workflow verbally. Operators run extraction using the configured templates and document sets, then inspect the returned values. In tools they had previously used, the next steps remained manual: download CSV or JSON, apply an ad hoc transformation, and push the result to a downstream process.

Page describes a low-code/no-code framework that lets operators build transformation and execution workflows into the same pipeline. This capability was described rather than demonstrated. Its role is consequential: the operator’s configuration must cover not only what to extract, but also how those results become usable inputs to the next system.

### Design for expertise, economics, and human review

[12:55](https://www.youtube.com/watch?v=08mH36_NVos&t=775s)

Making the sandbox available does not eliminate the need to develop skills. Vasanth recommends investing in domain experts’ prompt engineering because financial documents are difficult to describe precisely. Teams also need to understand what different LLM strategies do and how the available components fit their particular use case.

Prototype success is only one part of the production decision. The team must evaluate ROI and compare the custom AI application with an off-the-shelf product that might deliver the same job faster or at lower cost. A framework that reduces development effort does not automatically make every proposed application worthwhile.

**Design for human review first in highly regulated environments.** Vasanth ties that recommendation to compliance, regulation, and four-eyes checks. Human participation is therefore an explicit workflow requirement, even when a fully agentic process looks attractive.

Page then returns to the app factory, which was also explained rather than demonstrated. It combines the extraction templates, transformers, and executors refined in the sandbox into custom applications exposed through BlackRock’s application ecosystem. End users do not have to configure templates or work out downstream integration. They upload documents, run extraction, and start the complete pipeline. The expert’s configuration becomes the user’s application.

### Where the framework’s scope ends

[15:23](https://www.youtube.com/watch?v=08mH36_NVos&t=923s)

The first audience question extends the workflow from IPO processing and ETL toward executive analytics. It asks how instrument-level features such as term, maturity, and duration could become higher-level asset-and-liability information, with decentralized data in mind. The suggestion that a CEO would be the main user is the questioner’s premise, not a deployment described by the presenters.

Vasanth narrows the scope: the framework targets investment operations domain experts who are building applications. An executive memo or an asset-and-liability analysis would be a different initiative, which might or might not use this framework. Components can be reused, but the answer does not specify an architecture for aggregating instrument features into executive analytics.

The final question comes from an insurance practitioner facing similar document-processing problems. How should extraction evaluations account for failures that begin with OCR, or for models that misunderstand domain terminology despite prompting? This asks about correctness across the processing chain, not simply whether a result satisfies its output schema.

Vasanth’s response addresses information security and boundaries: controls and policies apply at the infrastructure, platform, application, and user layers within the organization’s network. Those controls address security concerns, but they do not establish whether an extracted field matches its source document.

Page adds that the team selects among multiple providers, strategies, and engineering adjustments according to the use case, rather than applying one RAG recipe everywhere. The closing answer does not provide a concrete evaluation dataset, OCR test, extraction metric, or accuracy result. The framework makes expert iteration and application delivery more repeatable; the question of how to measure extraction correctness remains open in this account.

## Resources

- [Introducing GPT-4.1 in the API](https://openai.com/index/gpt-4-1/) — April 2025 announcement documenting one-million-token context windows and the model family's long-context capabilities.
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) — Original RAG research combining language generation with retrieved passages from a dense document index.
- [Context is King: Scaling Beyond Prompt Engineering at BlackRock](https://mlopsworld.com/speakers/) — Conference abstract from the same speakers covering context construction, evaluation, and financial AI applications.

## Slides

### Accelerating Investment Operations: How BlackRock Builds Custom…

BlackRock data team members Infant Vasanth and Vaibhav Page introduce their talk on scaling custom AI and knowledge apps for investment operations.

[0:34](https://www.youtube.com/watch?v=08mH36_NVos&t=34s) [BlackRock title slide naming presenters Vaibhav Page and Infant Vasanth, with a small stage video inset.](https://assets.ai.engineer/sites/aiecode2025/sha256/ad2e45c1a294111eebb0eeff6caec8704979afec2db826b1a5810ad88658e37e.webp)

Accelerating Investment Operations: How BlackRock Builds Custom Knowledge Apps at Scale.

BlackRock®

Vaibhav Page
Principal Engineer

Infant Vasanth
Senior Director of Engineering

AIE
aws
World’s Fair

### Accelerating Investment Operations

BlackRock’s investment operations teams build complex internal tools for their domains, making rapid app delivery a priority.

[1:17](https://www.youtube.com/watch?v=08mH36_NVos&t=77s) [BlackRock title slide about building custom knowledge apps at scale, with presenter names and roles.](https://assets.ai.engineer/sites/aiecode2025/sha256/e352bc20ac548d67ff2e6ec9907219948246fbff46e0b0a9608cceb4eaf9ee2d.webp)

Accelerating Investment Operations: How BlackRock Builds Custom Knowledge Apps at Scale.

BlackRock®

Vaibhav Page
Principal Engineer

Infant Vasanth
Senior Director of Engineering

AIE
aws

### Enterprise Use Cases

BlackRock maps four custom app patterns: document retrieval and extraction, parallel extraction and synthesis, knowledge-graph Q&A with sources, and agents that interact and use tools.

[1:30](https://www.youtube.com/watch?v=08mH36_NVos&t=90s) [Four columns depict document extraction, automation, knowledge search, and data agent applications.](https://assets.ai.engineer/sites/aiecode2025/sha256/d4f302349074a7039963a6eb82ebc4a92d8ea03218b12a223d88fb84637b08cd.webp)

ENTERPRISE USE CASES

Document Extraction & Processing
Embeddings
Retrieval + LLM calls
Data Model Mappings
CUSTOM APP
DATA EXTRACTION APPS

Automation & Workflows
Data Source Events
Document
Document
Document
Extraction
Extraction
Extraction
Synthesis
CUSTOM APP
NEW ISSUE OPS APP

Knowledge Search & Q/A
Knowledge Graph DB
Source:
Entity:
Answer:
CUSTOM APP
DATA QC COPILOT

Data Agents
Tools
Tools
CUSTOM APP
DATA AGENT APPS

Fig 1.1 Enterprise Use Cases

Interface text:
AIE

### New Issue Operations

A simplified workflow extracts structured security data from prospectuses and term sheets, then sends it through validation to downstream applications. Operators provide templates and approvals; engineering teams handle transformations.

[2:20](https://www.youtube.com/watch?v=08mH36_NVos&t=140s) [A workflow routes prospectuses and term sheets through extraction, structured output, validation, and downstream applications.](https://assets.ai.engineer/sites/aiecode2025/sha256/51a6051bb97a594826a3b639930cfbba23d99d3d3344e6c784e6aa355233e362.webp)

USE CASE – NEW ISSUE OPERATIONS

Operators / SMEs
Approve / Override
Extraction Templates
Transformations
Engineering Teams
Executors

Prospectus
Termsheets
Document Extraction Pipeline

Structured Output
Issuer: "ABC"
Maturity Date: 2025-01-01
Call Price: 100
Ratings: BBB-
Call Date: 2025-01-01
Structure: Senior
LEI: ABCD

Validation / Feedback
Downstream Applications

Fig 1.2 New Asset Issuance Operations Lifecycle

Interface text:
AIE
aws

[2:55](https://www.youtube.com/watch?v=08mH36_NVos&t=175s) [Diagram showing document extraction, structured output, validation, and downstream applications.](https://assets.ai.engineer/sites/aiecode2025/sha256/fa68af275bed610c039aa5ed289d6ed8320a54ab1cb5c65ef2923ad9f408911c.webp)

USE CASE – NEW ISSUE OPERATIONS

Operators / SMEs
Approve / Override
Extraction Templates
Transformations
Engineering Teams
Executors

Prospectus
Termsheets
Document Extraction Pipeline

Structured Output
Issuer: "ABC"
Maturity Date: 2025-01-01
Call Price: 100
Ratings: BBB-
Call Date: 2025-01-01
Structure: Senior
LEI: ABCD

Validation / Feedback
Downstream Applications

Fig 1.2 New Asset Issuance Operations Lifecycle

Interface text:
AIE
Microsoft
smol ai

### Challenges with Scaling AI Knowledge Apps

BlackRock’s extraction prompts grow with instrument complexity. Teams must iterate and evaluate prompts, choose strategies for different documents, and deploy apps with access, resource, and cost controls.

[3:58](https://www.youtube.com/watch?v=08mH36_NVos&t=238s) [Three columns list challenges in prompt engineering, LLM strategies, and app deployment.](https://assets.ai.engineer/sites/aiecode2025/sha256/32c506ff6523253872f473e4d0881749fa59953bd71593febbb0e84892342b50.webp)

CHALLENGES WITH SCALING AI KNOWLEDGE APPS

Prompt Engineering
• Domain Expertise
• Managing & Iterating Prompts
• Version & Compare Prompts
• Evaluating Prompt Quality

LLM Strategies
• RAG vs MoE vs CoT Strategies
• Bad Retrieval, Low Precision
• Bad Response, Hallucination, Irrelevance
• Context & Model Limitations

App Deployment
• Domain Federation
• App Distribution & Access Control
• Resource Management, Security
• Model Latency, Cost

Table 1.1 AI Knowledge App Scale Challenges

AIE
aws
World's Fair

### DocuFlow Architecture

DocuFlow’s sandbox addresses app-development bottlenecks with prompt templates, LLM strategies, extraction runs, and transformer and executor logic. The API sits above orchestration workflows.

[7:01](https://www.youtube.com/watch?v=08mH36_NVos&t=421s) [Architecture diagram connecting a data platform, sandbox, app factory, API, orchestration layer, and developer platform.](https://assets.ai.engineer/sites/aiecode2025/sha256/82e93d60546779b3aad5e2afedf0a44590047dc4f903523eb7c59c215535b3d0.webp)

ARCHITECTURE

DATA PLATFORM
Data Sourcing
Metadata Enrichment
Embeddings
Post Processing

SANDBOX
Prompt / Extraction Templates
LLM / Extraction Strategies
Extraction Runs & Evals
Transformers & Executors

APP FACTORY

DOCUFLOW API

ORCHESTRATION LAYER
Embeddings Workflows
Extraction Workflows
Transformer Workflows
Executor Workflows

DEVELOPER PLATFORM
Apps
Functions
Governance & Workload Risk Profiles Management

Fig 1.3 DocuFlow Architecture

Interface text:
AIE
World's Fair

[7:09](https://www.youtube.com/watch?v=08mH36_NVos&t=429s) [DocuFlow architecture showing the data platform, sandbox, app factory, API, orchestration workflows, and developer platform.](https://assets.ai.engineer/sites/aiecode2025/sha256/b509af6c7e4be92fcf179ed378e8a9677d46222c7d414f7042831a33b8070a12.webp)

ARCHITECTURE

DATA PLATFORM
Data Sourcing
Metadata Enrichment
Embeddings
Post Processing

SANDBOX
Prompt / Extraction Templates
LLM / Extraction Strategies
Extraction Runs & Evals
Transformers & Executors

APP FACTORY

DOCUFLOW API

ORCHESTRATION LAYER
Embeddings Workflows
Extraction Workflows
Transformer Workflows
Executor Workflows

DEVELOPER PLATFORM
Apps
Functions
Governance & Workload Risk Profiles Management

Fig 1.3 DocuFlow Architecture

Interface text:
AIE
World's Fair

### Docuflow extraction templates

In a slimmed-down demo, Docuflow’s sandbox lists three extraction templates. The speaker describes using the sandbox to refine templates, run extractions on document sets, and compare results.

[8:37](https://www.youtube.com/watch?v=08mH36_NVos&t=517s) [Docuflow displays a list of three extraction templates, with navigation and template management controls.](https://assets.ai.engineer/sites/aiecode2025/sha256/4150b8ad1e664f5363017d0a93c8cff2f90b83ab4e64b6f1db9609674c50f68e.webp)

Extraction Templates

Create Extraction Template
Refresh Extraction Templates
Showing: 3 Results
Sort By: Default
Search

Name | Description | Fields | Created By | Updated At | Created At | Actions
Prospectus Analyzer | Extraction template for analyzing pros... | 4 | vpage | Jun 04, 2025, 12:18 AM | Jun 04, 2025, 03:19 AM
Private Credit Deal Template | Template for private credit deal extrac... | 3 | vpage | Jun 04, 2025, 12:03 AM | Jun 04, 2025, 12:02 AM
LPA template | Template for LPA extractions | 1 | vpage | Jun 04, 2025, 12:03 AM | Jun 04, 2025, 12:02 AM

Interface text:
Docuflow
Studio Project: hello
Sandbox
Extraction Templates
Document Sets
Extractions
Workflows
Factory
Apps
by BlackRock
Dark

Browser tabs: Docuflow; Docuflow App; New Tab
localhost:7080/extraction-templates
AIE

### Docuflow Extraction Templates

Docuflow’s sandbox lists three extraction templates. The speaker says operators define fields, prompts and data types, and need checks, constraints and dependencies between fields.

[9:03](https://www.youtube.com/watch?v=08mH36_NVos&t=543s) [Docuflow displays three extraction templates and controls for managing them.](https://assets.ai.engineer/sites/aiecode2025/sha256/58d9f12a1bc46d205069286863c6679ea6454b08b76b3a138705b368a3d6def2.webp)

Extraction Templates

Showing: 3 Results

Name | Description | Fields | Created By | Updated At | Created At | Actions
Prospectus Analyzer | Extraction template for analyzing pros... | 4 | vpage | Jun 04, 2025, 12:18 AM | Jun 04, 2025, 03:19 AM
Private Credit Deal Template | Template for private credit deal extrac... | 3 | vpage | Jun 04, 2025, 12:03 AM | Jun 04, 2025, 12:02 AM
LPA template | Template for LPA extractions | 1 | vpage | Jun 04, 2025, 12:03 AM | Jun 04, 2025, 12:02 AM

Interface text:
Docuflow
Studio Project: hello
Create Extraction Template
Refresh Extraction Templates
Sort By: Default
Search
Sandbox
Extraction Templates
Document Sets
Extractions
Workflows
Factory
Apps
by BlackRock
Dark
localhost:7080/extraction-templates
Docuflow
Docuflow App
New Tab
AIE
World's Fair
aws

[9:07](https://www.youtube.com/watch?v=08mH36_NVos&t=547s) [Docuflow lists three extraction templates with field counts and actions.](https://assets.ai.engineer/sites/aiecode2025/sha256/1667ed80d90af9c2bec76836774b8b71640adbd2238a170ad95300b7bca1fd51.webp)

Extraction Templates

Showing: 3 Results

Name | Description | Fields | Created By | Updated At | Created At | Actions
Prospectus Analyzer | Extraction template for analyzing pros... | 4 | vpage | Jun 04, 2025, 12:18 AM | Jun 04, 2025, 03:19 AM
Private Credit Deal Template | Template for private credit deal extrac... | 3 | vpage | Jun 04, 2025, 12:03 AM | Jun 04, 2025, 12:02 AM
LPA template | Template for LPA extractions | 1 | vpage | Jun 04, 2025, 12:03 AM | Jun 04, 2025, 12:02 AM

Interface text:
Docuflow
Studio Project: hello
Create Extraction Template
Refresh Extraction Templates
Sort By: Default
Search
Sandbox
Extraction Templates
Document Sets
Extractions
Workflows
Factory
Apps
by BlackRock
Dark
localhost:7080/extraction-templates
Docuflow
Docuflow App
New Tab
AIE
World's Fair
Microsoft
smol ai

### Template Editor

The sample groups Callable, Call Price, and Call Date; the speaker explains that callable bonds can require values for the latter two.

[10:23](https://www.youtube.com/watch?v=08mH36_NVos&t=623s) [The template editor shows issuer and call-detail fields with their types and requirements.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a1af196c9190d2b21a24c2fa14ec41528910966516cf44ed6a1d0f934f8a2fa.webp)

Template Editor

Template Name: Prospectus Analyzer
Description: Extraction template for analyzing prospectuses
System Prompt: You are expert analyzing investment grade corporate bonds
Edit Template

Issuer — Containing 1 Fields
1: Issuer
An issuer is a legal entity that develops, registers, and sells securities to finance its operations. Clearly extract the o...
Extracted | String | Required

Call Details — Containing 3 Fields
1: Callable
"Yes" if issuer can redeem partly or wholly before maturity. Return "C" if callable, otherwise "Not Available".
Extracted | String | Required
2: Call Price
Price at which issuer may redeem callable securities. Default to 100 if callable but not stated; blank if not callable.
Extracted | Number | Required
3: Call Date
Extracted | Date | Required

Interface text:
Docuflow
Studio Project: hello
Sandbox
Extraction Templates
Document Sets
Extractions
Workflows
Factory
Apps
by BlackRock
Dark
Screen Recording 2025-06-04 at 9.10.42 AM.mov
00:07 / 01:31
AIE
World's Fair
Microsoft
smol ai

### Field Configuration

The demo configures a field’s name, type, source, description and required status. The speaker explains that a field can be extracted or derived through a downstream transformation.

[10:33](https://www.youtube.com/watch?v=08mH36_NVos&t=633s) [A field configuration dialog shows name, data type, source, description and required settings.](https://assets.ai.engineer/sites/aiecode2025/sha256/273ad0416218a160d6d6bf5ebf6f71fed7fec0cad17f890d1b4b72860fab1dca.webp)

Field Configuration

ABCD

Name: *
ABCD

Data Type: *
String

Field Source: *
Extraction via LLM or Programmatically Generated?
Extracted
Extraction Prompt

Description:
The first date on which the issuer is permitted to call (redeem) the bond prior to maturity. Only applicable if the bond is callable.

Always Required? *
Select

Cancel
Save New Field

Interface text:
Docuflow
Studio Project: hello
Sandbox
Extraction Templates
Document Sets
Extractions
Workflows
Factory
Apps
Save Prompt
Cancel
Delete Field
by BlackRock
Dark
AIE
aws
World’s Fair
00:16 / 01:31

### Field Configuration

The demo opens an extraction-prompt editor with a maturity-date example. The field form also shows required status, dependency controls, and allowed values for restricted fields.

[10:40](https://www.youtube.com/watch?v=08mH36_NVos&t=640s) [An extraction-prompt editor provides guidance and an example prompt for a financial field.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce3bab4d69eb3aadba3d8e41ad80967df483f62e1232abbbc04102e98a9e5762.webp)

Edit Extraction Prompt

Enter a prompt that tells the model how to extract a specific financial field from documents. Include what to look for, keywords or phrases that may indicate the field, typical formats (e.g., percentages, dates, dollar amounts), and any rules for selecting between multiple values.

Extract the bond's maturity date. Look for terms like "maturity", "matures on", or "due date". The date format may vary (e.g., January 15, 2030 or 01/15/30). Return the date in YYYY-MM-DD format.

Interface text:
Field Configuration
Docuflow
Sandbox
Extraction Templates
Document Sets
Extractions
Workflows
Factory
Apps
Save Prompt
Cancel
Delete Field
Save New Field
Studio Project: hello
by BlackRock
Dark
Screen Recording 2025-06-04 at 9.10.42 AM.mov
00:17 / 01:31
AIE
World's Fair
Microsoft
smol ai

[10:53](https://www.youtube.com/watch?v=08mH36_NVos&t=653s) [Docuflow field configuration form showing data type, source, required status, and dependency controls.](https://assets.ai.engineer/sites/aiecode2025/sha256/84034e95a87365b7530b529a9dd4f3b0a1b38abbd802cdb3fed21978bece66ae.webp)

Field Configuration

ABCD
Name: *
ABCD
Data Type: *
String
Field Source: *
Extraction via LLM or Programmatically Generated?
Extracted
Extraction Prompt
Description:
The first date on which the issuer is permitted to call (redeem) the bond prior to maturity. Only applicable if the bond is callable.
Always Required? *
Yes
Add Field Dependency:
Select Dependent Field
Select State of Field
Select Value
Selected Field
ABCD
Field: Select
Select
For Restricted Fields, define allowed values here:
Clear All
Cancel
Save New Field

Interface text:
Docuflow
Sandbox
Extraction Templates
Document Sets
Extractions
Workflows
Factory
Apps
Save Prompt
Cancel
Delete Field
Studio Project: hello
by BlackRock
Dark
Screen Recording 2025-06-04 at 9.10.42 AM.mov
00:21 / 01:31
AIE
World's Fair
aws

### Document sets

Docuflow lists three document sets. The speaker says documents are ingested from the data platform, tagged by business category, labeled, and embedded.

[11:07](https://www.youtube.com/watch?v=08mH36_NVos&t=667s) [Docuflow document sets page showing a table of three example sets and controls to create or refresh sets.](https://assets.ai.engineer/sites/aiecode2025/sha256/6a4625ee4259bbefe1eda0ab313cf2cbc29bb563f002b7cff0016afbdc323785.webp)

Document Sets

Create Document Set
Refresh Document Sets
Showing: 3 Results
Sort By: Default
Search

Name | Description | Created On | Updated On | Actions
ISSUER 1 | Test Issuer | 2025-06-04 00:04:28 | 2025-06-04 00:04:28
TEST ISSUER A | TEST ISSUER A | 2025-06-04 00:06:25 | 2025-06-04 00:06:25
TEST DOCUMENT SET | TEST DOCUMENT SET | 2025-06-04 00:06:41 | 2025-06-04 00:06:41

Interface text:
Docuflow
Studio Project: hello
Sandbox
Extraction Templates
Document Sets
Extractions
Workflows
Factory
Apps
by BlackRock
Dark
QuickTime Player
File  Edit  View  Window  Help
Screen Recording 2025-06-04 at 9.10.42 AM.mov
00:31 / 01:31
AIE
World's Fair
Microsoft
smol ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=08mH36_NVos&t=0s) [on-hold music] Hi, everyone.

[0:15](https://www.youtube.com/watch?v=08mH36_NVos&t=15s) Thank you for having us. I'm Infant, Director of Engineering at BlackRock. This is my colleague, Vaibhav, Principal Engineer, and we both work for the data teams at BlackRock. And today, we are gonna talk about how we can scale building custom applications in BlackRock.

[0:28](https://www.youtube.com/watch?v=08mH36_NVos&t=28s) Specifically, we are talking about, like, AI applications and knowledge apps at BlackRock. Right. So just to level set before I get into the, uh, details. So BlackRock is an asset management firm, the world's largest asset manager.

[0:39](https://www.youtube.com/watch?v=08mH36_NVos&t=39s) What we do is our portfolio managers, analysts get a torrent of information on a daily basis. They synthesize this information, they develop an investment strategy, and then they rebalance their portfolios, uh, which ultimately results in a particular trade.

[0:53](https://www.youtube.com/watch?v=08mH36_NVos&t=53s) Now, the investment operations teams, you can think of that as the teams that are the backbone or the engine that make sure that all of the activities that the investment managers actually perform on a day-to-day basis, like, runs smoothly, right?

[1:06](https://www.youtube.com/watch?v=08mH36_NVos&t=66s) So these teams are kind of responsible for, like, a-acquiring the data that you kind of need, right, uh, to actually executing a trade, running through compliance, all the way to, like, all of the post-trading activities, right?

[1:17](https://www.youtube.com/watch?v=08mH36_NVos&t=77s) So all of these teams actually have to build these internal tools that are actually fairly complex for each of their domains, right? So building apps and pushing out these apps, uh, relatively quickly is, like, uh, of utmost important to us, right?

[1:31](https://www.youtube.com/watch?v=08mH36_NVos&t=91s) So if you move on to the next slide. Again, if you actually classify what kind of apps we are talking about, what you'll see is it kind of falls into, like, four different buckets, right?

[1:39](https://www.youtube.com/watch?v=08mH36_NVos&t=99s) One is everything to do with document extraction. So I have an app, I kind of want to, like, extract entities out of it in that bucket. Second has to do everything with, like, "Hey, I kind of want to define a complex, uh, workflow or an automation."

[1:51](https://www.youtube.com/watch?v=08mH36_NVos&t=111s) So I could have a case where I kind of want to run through X number of steps and then integrate to my downstream systems. And then you have the normal, like, Q&A-s type systems that you look at, like, this is your chat interfaces, and finally, like, the, the agentic systems, right?

[2:05](https://www.youtube.com/watch?v=08mH36_NVos&t=125s) So in each of these domains, what we see is, um, we have this, like, big opportunity to leverage your models and LLMs to either augment our existing systems, uh, or like, uh, kind of like supercharge those, right?

[2:19](https://www.youtube.com/watch?v=08mH36_NVos&t=139s) So that, that is, like, the domain we are speaking about. So I'll move quickly to one particular use case. So this, this is a use case that came to us, like, about, like, three to four months back, right?

[2:30](https://www.youtube.com/watch?v=08mH36_NVos&t=150s) And we have a team within the investment operations space. It's known as the new issue operations team, right? So this team is kind of responsible for setting up securities, uh, whenever there is, like, a market event, right?

[2:41](https://www.youtube.com/watch?v=08mH36_NVos&t=161s) So a company goes IPO or, like, there is, like, a stock split for a particular organization, right? The team actually has to take the security, and they have to set it up in our internal systems before our portfolio managers or traders can actually action upon it, right?

[2:55](https://www.youtube.com/watch?v=08mH36_NVos&t=175s) So we kind of have to build this tool for the investment operations team, right, to set up a particular security. This is like-- Actually, honestly, this is, like, a super simplified version of what happens.

[3:05](https://www.youtube.com/watch?v=08mH36_NVos&t=185s) But at super high level, we have to build an app that is able to, like, ingest your prospectus or a term sheet. It pushes it through a particular pipeline, right?

[3:14](https://www.youtube.com/watch?v=08mH36_NVos&t=194s) Uh, then you talk to your domain experts, and these are, like, your, uh, business teams, your equity teams, ETF teams, et cetera. They actually n-know how to set up these complex instruments.

[3:23](https://www.youtube.com/watch?v=08mH36_NVos&t=203s) You get some kind of structured output. And now that team works with the engineering teams to actually build this transformation logic and the like, and then integrate it with our downstream applications.

[3:33](https://www.youtube.com/watch?v=08mH36_NVos&t=213s) So you can see that this process actually takes a long time, right? So building an app, and then you're introducing new model providers, you're trying to put in, like, new strategies.

[3:42](https://www.youtube.com/watch?v=08mH36_NVos&t=222s) The-- lot of challenges to get a single app out, right? We tried this with agentic systems. Doesn't quite work right now because of the complexity and the, the domain knowledge that's imbued in the human head, right? [laughs]

[3:56](https://www.youtube.com/watch?v=08mH36_NVos&t=236s) So the big challenges with scale are, again, these three categories, right? One is we are spending a lot of time with our domain experts prompt engineering, right? So in the first phase where we have to extract these documents, right,

[4:11](https://www.youtube.com/watch?v=08mH36_NVos&t=251s) they're very complex, right? Your prompt itself, uh, in our simplest case, like, started with, like, a couple of sentences. Before you knew it, you're trying to describe this financial instrument, and it is, like, three paragraphs long, right?

[4:22](https://www.youtube.com/watch?v=08mH36_NVos&t=262s) Uh, so it, it-- There's this challenge of, like, "Hey, I have to iterate over these prompts. I have to version and compare these prompts. How do I manage that effectively?"

[4:29](https://www.youtube.com/watch?v=08mH36_NVos&t=269s) And I think even the previous speaker had mentioned you kind of need to eval and have this data set. How, how good is your prompt performing? So that's the first set of challenges in creating, like, AI apps itself, right?

[4:38](https://www.youtube.com/watch?v=08mH36_NVos&t=278s) How are you gonna manage this? In what direction? Second set of challenges is around, like, LLM strategies, right? What I mean by this is, like, when you're building an, uh, um, AI app, so to speak, you have to choose what strategy.

[4:52](https://www.youtube.com/watch?v=08mH36_NVos&t=292s) Uh, am I gonna use, like, a RAG-based approach, right? Or am I going to use a chain of thought based approach? Even for a simple task of, like, data extraction.

[4:59](https://www.youtube.com/watch?v=08mH36_NVos&t=299s) Depending on what your instrument is, this actually varies, uh, very highly, right? If you take, like, an investment corporate bond, like, the vanilla one, it's fairly simple. I can do this with, like, in context, pass it to model.

[5:11](https://www.youtube.com/watch?v=08mH36_NVos&t=311s) I'm able to get my stuff back if the document size is small, right? Some documents are, like, thousands of pages long, ten thousand pages long. Now suddenly you're like, "Oh, okay.

[5:19](https://www.youtube.com/watch?v=08mH36_NVos&t=319s) I don't know if I can pass, uh, more than a million tokens into, say, uh, the OpenAI models. What do I do then?" Right? Then okay, I need to choose a different strategy.

[5:28](https://www.youtube.com/watch?v=08mH36_NVos&t=328s) And often what we do is we have to choose different strategies and kind of mix them with your prompts to kind of build this iterative process where, like, I have to play around with my prompts.

[5:38](https://www.youtube.com/watch?v=08mH36_NVos&t=338s) I have to play around with the different LLM strategies, and we kind of ma- want to make that process as quickly as possible. That's a challenge, right? Then you have obviously the context limitations, model limitations, different vendors, and you're trying and testing, uh, things, uh, uh, for quite a while.

[5:52](https://www.youtube.com/watch?v=08mH36_NVos&t=352s) And this g- kind of goes into the month, right? Then the biggest challenge is, like, okay, fine, I've kind of built this app. Now what? How do I get this to deployment?

[6:00](https://www.youtube.com/watch?v=08mH36_NVos&t=360s) And it's this whole other set of challenges, right? You have your traditional challenges which just has to do with distribution, access control. How am I going to federate the app to the users?

[6:09](https://www.youtube.com/watch?v=08mH36_NVos&t=369s) But then in the AI space, it's like you have this new challenge of, like, what type of cluster am I going to deploy this to, right? So our equity team would come and say something like, "Hey, I need to analyze, you know, five hundred research reports, like, o- overnight.

[6:22](https://www.youtube.com/watch?v=08mH36_NVos&t=382s) Can you help me do this?" Right? So, okay, if you're going to do that, I probably have to have, like, a GPU-based inference cluster that I can kind of spin up, right?

[6:30](https://www.youtube.com/watch?v=08mH36_NVos&t=390s) This is the use case that I kind of described with this, the new issue setup. In that case, what we do is, okay, I don't really want to use my GPU inference cluster, et cetera.

[6:38](https://www.youtube.com/watch?v=08mH36_NVos&t=398s) What I do instead is I use, like, a burstable cluster, right? So all those have to be kind of, like, uh, defined so that our app deployment phase is, like, as close to, like, a CI/CD pipeline as possible.

[6:51](https://www.youtube.com/watch?v=08mH36_NVos&t=411s) Then you have, like, cost controls. So these are-- Again, it's not an exhaustive list. I think I'm-- what I'm trying to highlight is the challenges with kind of building AI apps, right?

[7:00](https://www.youtube.com/watch?v=08mH36_NVos&t=420s) So what we did at BlackRock is-- What I'm gonna do is I'll kind of give you a high-level architecture, uh, and then maybe Vaibhav, you can dive into the details and mechanics of how this works and how we are able to build apps relatively quickly, right?

[7:14](https://www.youtube.com/watch?v=08mH36_NVos&t=434s) We're able to-- We took this, uh, an app took us close to, like, eight months to-- somewhere between three to eight months to build a single app for a complex use case, and we're able to compress time, bring it down to, like, a couple of days, right?

[7:25](https://www.youtube.com/watch?v=08mH36_NVos&t=445s) We achieved that by building out this framework. Uh, what I kind of want to focus on is on the top two boxes that you see, which is your sandbox and your app factory, right?

[7:34](https://www.youtube.com/watch?v=08mH36_NVos&t=454s) So to the, uh, the data platform and the developer platform, it's, like, the name suggests, hey, platform is responsible for ingesting data, et cetera, right? You have an orchestration layer that has a pipeline that kind of, like, transforms it, brings it into some, uh, new format, and then you kind of distribute that as a app or

[7:50](https://www.youtube.com/watch?v=08mH36_NVos&t=470s) a report. What kind of accelerates at app development is, like, if you're able to federate out those pain points or those bottlenecks, which is, like, prompt creation or extraction templates, choosing an LLM strategy, right, having extraction plans or, like, and then building out these logic pieces which are calling transformer and executors.

[8:11](https://www.youtube.com/watch?v=08mH36_NVos&t=491s) If you can get that sandbox out into the hands of the domain experts, then your iteration speed becomes really fast, right? So you're kind of saying that, "Hey, I have this modular component.

[8:20](https://www.youtube.com/watch?v=08mH36_NVos&t=500s) Can I move across the situation really quickly and then pass it along to an app factory?" Which is like our cloud-native operator which takes a definition and spits out an app, right?

[8:28](https://www.youtube.com/watch?v=08mH36_NVos&t=508s) So that's super high level. With that, quick demo. [chuckles]

[8:33](https://www.youtube.com/watch?v=08mH36_NVos&t=513s) Perfect. All right. Cool. So what I'm gonna show you guys is pretty slimmed down version of the actual tool we used internally. Um, so to start with, uh, when the operat-- So we have, like, two different, uh, concore components.

[8:50](https://www.youtube.com/watch?v=08mH36_NVos&t=530s) One is the sandbox, another one is the factory. So think of sandbox as a playground for the operators to sort of, like, quickly build and refine the extraction templates, uh, sort of run extraction on the set of documents, and then compare and contrast the results of these extractions.

[9:05](https://www.youtube.com/watch?v=08mH36_NVos&t=545s) Um, [clears throat] so it's sort of like to get started with the extraction template itself, uh, you might have seen in the other tools, both closed and open source, they have similar concept like prompt template management.

[9:16](https://www.youtube.com/watch?v=08mH36_NVos&t=556s) Where you have certain fields that you want to extract out of the documents and you have their corresponding prompts and some metadata that you can associate with them, such as the data type that you expect of the, the final result values.

[9:27](https://www.youtube.com/watch?v=08mH36_NVos&t=567s) But when these operators sort of, like, trying to run extractions on these documents, they need far more sort of, like, greater configuration capabilities than just, like, configuring prompts and configuring the data types that they expect for the end result.

[9:40](https://www.youtube.com/watch?v=08mH36_NVos&t=580s) So they need like, "Hey, I need to have multiple QC checks on the result values. I need to have a lot of validations and constraints on the fields." And there might be, like, inter-field dependencies, uh, what, what the fields that are getting extracted.

[9:54](https://www.youtube.com/watch?v=08mH36_NVos&t=594s) So as Infant mentioned with the new se- security operation, um, issuance basically onboarding that stuff, there could be a case where, uh, the security or the bond is callable and you have other fields such as call date and call price which now needs to have a value.

[10:10](https://www.youtube.com/watch?v=08mH36_NVos&t=610s) So there is like this inter sort of like field dependencies that operators sort of like, uh, need to... They, they need to take that into consideration, be able to configure that.

[10:18](https://www.youtube.com/watch?v=08mH36_NVos&t=618s) So here is like what a, like a sample, uh, extraction template looks like.

[10:24](https://www.youtube.com/watch?v=08mH36_NVos&t=624s) So here is how a-- Again, this is a, a example template where we have like issuer, callable, call price, and call date, these fields set up. And to sort of like add new fields, we will define the field name, uh, define the data type that is expected out of that, uh, define the source, whether it's extracted or

[10:41](https://www.youtube.com/watch?v=08mH36_NVos&t=641s) derived. Not every time you wanna sort of like run an extraction for a field. There might be a derived field that operator expect which is sort of like, uh, populated through some transformation downstream.

[10:52](https://www.youtube.com/watch?v=08mH36_NVos&t=652s) Um, and once, uh, again, uh, whether the field is required and the field dependencies. Here is where you define what sort of like dependencies this field have and sort of validations, right?

[11:02](https://www.youtube.com/watch?v=08mH36_NVos&t=662s) So this is how they set up the extraction. The next thing is the document management itself. So this is where the documents are ingested, uh, from the, uh, the data platform.

[11:12](https://www.youtube.com/watch?v=08mH36_NVos&t=672s) They are tagged according to the business category, uh, and they are labeled, they are embedded, all of that stuff.

[11:19](https://www.youtube.com/watch?v=08mH36_NVos&t=679s) Okay. While-- I think while, uh, Vaibhav kind of brings it up. So I think, uh, what-- in essence, what we are saying is we have kind of built this tool which has like a UI component and like a framework that actually lets you take these different pieces and these modular components and give it to the hands of

[11:33](https://www.youtube.com/watch?v=08mH36_NVos&t=693s) like the domain expert to build out their app really quickly, right?

[11:38](https://www.youtube.com/watch?v=08mH36_NVos&t=698s) Look, uh, I think something-

[11:40](https://www.youtube.com/watch?v=08mH36_NVos&t=700s) Yeah

[11:40](https://www.youtube.com/watch?v=08mH36_NVos&t=700s) ... happened it just saying. So let me just sort of walk you guys the-- what happens next. Um, so like once you have set up the extraction templates and document set management, the operators basically run the extractions.

[11:50](https://www.youtube.com/watch?v=08mH36_NVos&t=710s) That's where they basically see the values that they expect from these documents and sort of like review them. Uh, the thing with-- we have seen with these operators trying to use other tools, uh...

[12:02](https://www.youtube.com/watch?v=08mH36_NVos&t=722s) No, this is the same. Um- Yeah, I do. Uh, the thing we have seen, uh, with these operators is that most of the tools, tools that they have used in past, uh, these tools basically does extraction.

[12:14](https://www.youtube.com/watch?v=08mH36_NVos&t=734s) Uh, they, they, they do a particular job at extraction. But when it comes to like, uh, "Hey, I need to now use this result that has been, uh, sh- presented to me and pass it to the downstream processes," the process right now is very manual, where they have to like download a CSV or a JSON file, do--

[12:29](https://www.youtube.com/watch?v=08mH36_NVos&t=749s) run manual or ad hoc transformation, and then push it to the downstream process. So what we have done, and again I can't show you, but what we have done is like build this sort of like lo- low-code, no-code framework where the operators can basically essentially, uh, run the, uh, sort of build this transformation and execution workflows and,

[12:50](https://www.youtube.com/watch?v=08mH36_NVos&t=770s) uh, sort of like have this end-to-end, uh, pipeline running. Uh, and-

[12:55](https://www.youtube.com/watch?v=08mH36_NVos&t=775s) I, I think, yeah. So I think we'll conclude by saying that our key takeaways of this, right? I would say there are like three key takeaways. Invest heavily on your like prompt engineering skills for your domain experts, especially in like the financial space and world.

[13:07](https://www.youtube.com/watch?v=08mH36_NVos&t=787s) Uh, defining and describing these documents is really hard, right? Uh, second is like educating the firm and the company on what an LLM strategy means, uh, and how to actually fix these different pieces for your particular use case.

[13:19](https://www.youtube.com/watch?v=08mH36_NVos&t=799s) And I think the third one I would say is, hey, uh, the key takeaway that we had is all of this is great in experimentation and prototyping mode, but if you kind of want to bring this, you have to really evaluate what your ROI is.

[13:30](https://www.youtube.com/watch?v=08mH36_NVos&t=810s) And as-- Is it gonna be like more expensive actually spinning up an AI app versus just having like an off-the-shelf, uh, product that does it quicker and faster, right?

[13:38](https://www.youtube.com/watch?v=08mH36_NVos&t=818s) So those are the three key takeaways in terms of like, uh, building apps at scale. And what we realized was like, hey, uh, this notion of like human in the loop.

[13:48](https://www.youtube.com/watch?v=08mH36_NVos&t=828s) And the one more thing I'll add is human in the loop, super important, right? We all are like really tempted, like let's go all agentic with this. Uh, but in the financial space with compliance, with regulations, you kind of need those four eyes check, and you kind of need the human in the loop.

[14:01](https://www.youtube.com/watch?v=08mH36_NVos&t=841s) So design for human in the loop first, uh, if you're in a highly regulated environment.

[14:05](https://www.youtube.com/watch?v=08mH36_NVos&t=845s) Yeah. And as Infant said, one thing we couldn't show is the whole app factory sort of like, uh, component, which is all the things that operators do through this iteration cycle of through the sandbox.

[14:16](https://www.youtube.com/watch?v=08mH36_NVos&t=856s) They take all that knowledge, the extraction templates, the transform it-- transformers, and executors to build through this workflow pipeline. And through our app, uh, ecosystem within, uh, BlackRock, they sort of like build this, uh, custom applications that are then exposed to the users, where users of this app don't have to worry about how to configure templates or

[14:36](https://www.youtube.com/watch?v=08mH36_NVos&t=876s) how to basically figure out how to integrate the result values into final downstream processes. They are presented with this whole end-to-end app, where they can just go and like sort of like-

[14:45](https://www.youtube.com/watch?v=08mH36_NVos&t=885s) Yeah

[14:45](https://www.youtube.com/watch?v=08mH36_NVos&t=885s) ... upload documents and run extraction and sort of, uh, get the whole pipeline set, uh, running.

[14:51](https://www.youtube.com/watch?v=08mH36_NVos&t=891s) Yeah. With that, we'll open up for questions. I think we have like a minute or two left.

[14:55](https://www.youtube.com/watch?v=08mH36_NVos&t=895s) Yeah.

[14:55](https://www.youtube.com/watch?v=08mH36_NVos&t=895s) Yeah. Um, so I have a question which may directly be related to...

[15:08](https://www.youtube.com/watch?v=08mH36_NVos&t=908s) Good morning.

[15:09](https://www.youtube.com/watch?v=08mH36_NVos&t=909s) Morning.

[15:10](https://www.youtube.com/watch?v=08mH36_NVos&t=910s) I have a question which may directly be related to the, uh, architecture that you developed.

[15:17](https://www.youtube.com/watch?v=08mH36_NVos&t=917s) Okay.

[15:18](https://www.youtube.com/watch?v=08mH36_NVos&t=918s) You can tell me, I can discuss later. But the question is going to be,

[15:23](https://www.youtube.com/watch?v=08mH36_NVos&t=923s) y- you have developed, uh, um, the key takeaways. One of those key takeaways had been in invest heavily on prompt engineering. So you have essentially automated the process from the leaf level.

[15:41](https://www.youtube.com/watch?v=08mH36_NVos&t=941s) For example, a company's coming to an IPO from that level all the way to cataloging through ETL processes, and then to finally to the data analytics. So now your CEO, who looks at the balance sheet assets and liability, will be using your AI the most.

[16:05](https://www.youtube.com/watch?v=08mH36_NVos&t=965s) And for C-- uh, your CEO, now what are the features involved here at the lowest level? For example, term, maturity, duration.

[16:16](https://www.youtube.com/watch?v=08mH36_NVos&t=976s) Yeah.

[16:16](https://www.youtube.com/watch?v=08mH36_NVos&t=976s) There are so many metrics at the leaf level. How are you transforming those features from the lowest level to highest level? I'm looking for an answer in reference to decentralized data-

[16:30](https://www.youtube.com/watch?v=08mH36_NVos&t=990s) Yeah

[16:30](https://www.youtube.com/watch?v=08mH36_NVos&t=990s) ... or-

[16:31](https://www.youtube.com/watch?v=08mH36_NVos&t=991s) I mean, I, I can give you a quick answer, and then we can discuss, uh, in detail like, uh, offline. I think, uh, real quickly, like the, the framework that we built was specifically targeting like the investment operation domain experts who are trying to build applications.

[16:43](https://www.youtube.com/watch?v=08mH36_NVos&t=1003s) To your question of like, "Hey, what does the CEO care about? Can I construct a memo that gives me my asset liabilities XYZ?" Uh, those would be like different initiatives which may or may not use our particular framework.

[16:54](https://www.youtube.com/watch?v=08mH36_NVos&t=1014s) Thank you.

[16:54](https://www.youtube.com/watch?v=08mH36_NVos&t=1014s) Um, but, uh, yes, there are many reusable components in here that people can use. Yeah.

[17:00](https://www.youtube.com/watch?v=08mH36_NVos&t=1020s) So I kind of wonder, you know, for something similar for each problem.

[17:11](https://www.youtube.com/watch?v=08mH36_NVos&t=1031s) Yeah. So I do like a lot of document processing for insurance company, pretty much same problems as you guys run into. So I wonder, how do you build evals around your information extraction from the documents, right?

[17:22](https://www.youtube.com/watch?v=08mH36_NVos&t=1042s) Because there are so many things that can go wrong, starting from OCR.

[17:26](https://www.youtube.com/watch?v=08mH36_NVos&t=1046s) Yeah.

[17:26](https://www.youtube.com/watch?v=08mH36_NVos&t=1046s) Like LLM doesn't understand what all these terms actually mean, no matter how you prompt it, right? All this stuff.

[17:32](https://www.youtube.com/watch?v=08mH36_NVos&t=1052s) And I think-

[17:33](https://www.youtube.com/watch?v=08mH36_NVos&t=1053s) So that's kind of what's-

[17:34](https://www.youtube.com/watch?v=08mH36_NVos&t=1054s) Yeah

[17:34](https://www.youtube.com/watch?v=08mH36_NVos&t=1054s) ... what the reason.

[17:34](https://www.youtube.com/watch?v=08mH36_NVos&t=1054s) Again, I mean, we had all of that that we wanted to show, but yeah. So it's, uh-

[17:39](https://www.youtube.com/watch?v=08mH36_NVos&t=1059s) I mean, I think a short answer-

[17:40](https://www.youtube.com/watch?v=08mH36_NVos&t=1060s) But I can... Yeah

[17:40](https://www.youtube.com/watch?v=08mH36_NVos&t=1060s) ... short answer to your question is in terms of like information security and what are the boundaries, uh, that we are putting in terms of like, hey, we are not having data leakage or errors or understanding of...

[17:50](https://www.youtube.com/watch?v=08mH36_NVos&t=1070s) Like, in terms of security, you can think of it as different layers, all the way from like your infra, platform, application, right, and the user levels. There are different controls and policies in place, uh, and it's also within your SD network.

[18:03](https://www.youtube.com/watch?v=08mH36_NVos&t=1083s) Like, I think there are policies across the stack that we can get into in detail later that kind of, uh, addresses your, um, concerns. Yeah.

[18:10](https://www.youtube.com/watch?v=08mH36_NVos&t=1090s) And also, also to your point, um, I think we have like different sort of like, uh, strategies that we use based on, uh, the sort of like the use case at hand.

[18:20](https://www.youtube.com/watch?v=08mH36_NVos&t=1100s) Uh, so it's not just like, hey, one RAG versus this. There are multiple model providers that we use, uh, multiple different strategies, et cetera, uh, different like, like engineering sort of tweaks.

[18:31](https://www.youtube.com/watch?v=08mH36_NVos&t=1111s) Uh, so it's a quite complex sort of, yeah, process.

[18:35](https://www.youtube.com/watch?v=08mH36_NVos&t=1115s) All right, very cool.

[18:36](https://www.youtube.com/watch?v=08mH36_NVos&t=1116s) Awesome.

[18:36](https://www.youtube.com/watch?v=08mH36_NVos&t=1116s) Thank you. All right. [audience applauding] [outro music]
