# BotDojo Launch: Enhancing AI Assistants with Evaluations and Synthetic Data

Paul Henry, BotDojo

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/PNjMwdCo_YM-botdojo-launch-enhancing-ai-assistants-evaluations) · [Video](https://www.youtube.com/watch?v=PNjMwdCo_YM)

## BotDojo Launch: Enhancing AI Assistants with Evaluations and Synthetic Data

Paul Henry’s BotDojo demo connects node-level traces, batch evaluations and documents generated from support conversations to test whether a chatbot has the information it needs.

Familiarity with LLM prompts and vector retrieval will help you follow the chatbot and document-generation flows.

### Getting beyond the weekend prototype

[0:15](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=15s)

Connecting a vector database to an LLM can take a weekend. How do you know the resulting chatbot is ready for customers? Paul Henry, founder of [BotDojo](https://www.botdojo.com/resources/about-us) and a former CTO, describes that gap from work with teams deploying LLM applications for hundreds of thousands of customers. The connection is easy; making the application dependable is harder.

The demonstration starts with a chatbot template Henry says customers already use. The proposed improvement combines **synthetic data** with **evaluations**: identify where the chatbot lacks information, make additional material available to retrieval, and run the questions again to see what changes.

### Following a question through the flow

[1:14](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=74s)

“How do I create a vector index in BotDojo?” is the first live question. As the chatbot runs, nodes light up in the editor: the flow takes the question, consults chat history, retrieves information from the vector database, and sends the resulting context to an AI model to produce an answer. This makes the retrieval-and-generation sequence visible rather than hiding it behind the chat interface.

Opening a node reveals the prompt sent to the LLM and its output. Henry also describes JSON Schema support when the selected model supports JSON output, mentioning Groq and **Claude**. That presentation-time description is conditional on model capabilities; it does not establish a particular provider API or strict schema-enforcement guarantee.

**Per-node traces** expose the exact data at each boundary, including what the retriever returned and what the LLM received. Those are different debugging questions: did retrieval supply useful evidence, and did the model use the supplied evidence well? The answer on screen includes an image and citations. Henry jokes, “We should ship it”—setting up the difference between an attractive answer and evidence that the chatbot works across questions.

![BotDojo support chatbot with a response and embedded screenshot on the left, connected workflow nodes on the right, and a green Check Passed indicator.](https://ai.engineer/api/talk-essay-media/76c72f577cbff5ea892b6a926ed8f888f5777ee83d228cace803e03c3f5a5152)

The support chatbot’s response includes an image, a footnote link, and a passed check beside the executed flow.

### Rerunning questions with a larger retrieval corpus

[2:21](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=141s)

BotDojo’s **Batches** feature runs collections of questions through a chatbot or other AI flow and evaluates the outputs. Henry opens an earlier run with five evaluations and attributes its red results to insufficient information in the vector database. The evaluations also check for hallucinations. Five evaluations does not establish how many test questions were in the batch.

To test the additional content, Henry changes the batch setup:

1. Clone the earlier batch and rename it “with generated data.”
2. Increase throughput to fit the live demonstration’s time limit.
3. Remove the index filter that excluded generated content from the earlier run.

The generated documents already exist; there is not enough time to create the full dataset on stage. **The substantive retrieval change is to admit previously excluded documents**, rather than regenerate the corpus during this rerun.

![Create Batch dialog showing evaluation checkboxes, concurrency control, model settings, an empty Index Filter editor, and Create And Run button.](https://ai.engineer/api/talk-essay-media/5ed7a62c98840f522fc23815dff6c9f163d4c4cf69974814b6542f10836ad668)

Batch settings with the index filter cleared for the run named “With generated data.”

A small TypeScript example makes that selection difference concrete. Here, `generated` records are present in the corpus in both cases; only their eligibility for retrieval changes:

```typescript
type DocumentRecord = {
  id: string;
  generated: boolean;
};

const corpus: DocumentRecord[] = [
  { id: "existing-document", generated: false },
  { id: "support-derived-document", generated: true },
];

const baselineCandidates = corpus.filter(doc => !doc.generated);
const expandedCandidates = [...corpus];
```

These illustrative records express the filter operation, not BotDojo’s filter syntax. In the demo, removing the filter lets the flow retrieve generated material alongside existing content.

| Run | Existing content | Generated content |
| --- | --- | --- |
| Earlier batch | Eligible | Filtered out |
| New batch | Eligible | Eligible |

Henry notes that model selection can also be changed for comparisons. He does not demonstrate a model change here; the comparison he sets up concerns access to generated content.

### Turning support conversations into documents

[3:26](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=206s)

While the batch runs, Henry opens the flow used to generate the synthetic data and starts an example run. It accepts multiple inputs, supplied here by pasting JSON from a previous run. The starting material is support tickets: conversations between live support agents and customers. Extracting their question-answer pairs supplies questions to send through the chatbot as test data.

The document-generation process then connects those support interactions to the existing knowledge base:

1. Retrieve relevant information from the existing index.
2. Use that information to write a document in the same writing style.
3. Evaluate inline whether the document contains enough information to answer the question.

The evaluation asks a specific question about the generated document’s usefulness. Producing plausible prose alone is not the completion criterion being checked.

A code node handles logic that would otherwise require a sprawling collection of low-code boxes. Henry describes TypeScript support and Python as coming soon at presentation time. The flow then writes the generated information into the vector index, making it available to the support chatbot when retrieval permits it. This closes the path from support conversations, through document generation and a sufficiency check, back into the corpus being tested.

![BotDojo document-generation workflow beside a generated response, with evaluation and vector-index nodes, a human-review warning, and a Completed notification.](https://ai.engineer/api/talk-essay-media/c3105c19dcd782ca928f1de9908ed32cb4d41a0b703cf39d96e0d56466a8652d)

The document-generation flow finishes with evaluation and vector-index steps visible.

### Waiting for the comparison

[4:45](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=285s)

Henry returns to the support chatbot to compare the earlier batch with the run that includes generated content, with roughly twenty seconds remaining. The comparison initially fails to appear, prompting a joke about having rehearsed it fifteen times. As the countdown continues, he adds an appeal for AI engineers to join the company and help fix it.

The comparison finally appears with a second to spare. Henry reports that the final comparison is all green, but supplies no numerical improvement, named final metric or test-question count. His closing description—“measurably improved something”—captures both the successful reveal and its limits. The demonstrated loop is concrete: evaluate a chatbot, admit additional support-derived documents into retrieval, and compare the rerun. The recording ends with that comparison, rather than a production deployment walkthrough.

## Resources

- [Run a Batch](https://docs.botdojo.com/docs/guides/build-an-agent-from-scratch/10-batches) — Current instructions for running a dataset through a Flow and comparing evaluation results after changes.
- [Add Evaluations](https://docs.botdojo.com/docs/guides/build-an-agent-from-scratch/9-setup-evals) — Configure code-based checks or LLM judges, map their inputs, and inspect scoring explanations.
- [Test and Inspect Traces](https://docs.botdojo.com/docs/guides/build-an-agent-from-scratch/8-test-qanda) — Inspect agent requests, node execution and retrieved evidence in the current BotDojo interface.
- [Code Nodes](https://docs.botdojo.com/docs/platform/build/flows/advanced/code-nodes) — Examples of adding custom logic to visual flows with schema-defined inputs, outputs and execution context.

## Slides

### Presenting Sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=5s) [Microsoft logo beneath the heading Presenting Sponsor.](https://assets.ai.engineer/sites/aiecode2025/sha256/934234db61120b061df74007ae76b4c00249761b65746cfb86309c8a6f76e39c.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and Platinum Sponsors

AWS is the innovation sponsor; MongoDB, Google Cloud, and Neo4j are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=10s) [AWS is listed as innovation sponsor; MongoDB, Google Cloud, and Neo4j are listed as platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b22a87f3e586edc6a549a82593dd5e988b6699477e7af74509fa0d360684c8e.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB®
Google Cloud
neo4j

### BotDojo Support Chat Bot Demo

The presenter opens a chatbot template. Its flow uses conversation history to form a retrieval query, then combines retrieved results and memory to answer questions and cite references.

[0:44](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=44s) [BotDojo flow list showing folders and several flows, including a support chatbot.](https://assets.ai.engineer/sites/aiecode2025/sha256/74c0a855c950b20fbfe7f9d2ab1bb42df07a92d1fb3aa7c9905609b66342b8e5.webp)

Flows

Agents
Lib
Daily Email on emerging issues
Generate Documents from Extracted Q&A
Review Release Notes and Suggest Doc Updates
Support Chat Bot
ZenDesk - Find Q&A from Chat Transcript

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/list/root/
BotDojo
AwesomeSauce > My Project > Flows
Delete
Copy
Move To Folder
New Folder
New Flow
Search
Project
Flows
Data
Batches
Feedback Queues
Analytics
Settings
Account
Projects
Integrations
Team
Name & Info

[0:59](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=59s) [Close view of a chatbot workflow with memory, question formulation, retrieval, and answer nodes.](https://assets.ai.engineer/sites/aiecode2025/sha256/eba21d5641347fbac6fae8abf84dd23b905261ee1386255f647ccd80458d6f86.webp)

Conversational AI Assistant

This example implements Retrieval-Augmented Generation (RAG) and then cites the references.
Includes:
• Conversation Memory
• Citations
• RAG Evaluations

Start 0
question

Buffer Memory 0
Number of Messages: 10
Memory Key: buffer_memory
User Message
Memory

Special Instructions
Text Output
special_instructions

Formulate a question from the conversation history.

Former History
question
history
question

Form new question based on q...
Function Configuration: Function
Memory
history
question
combined_question

Retriever 0
Index: [illegible]
Number of Results: 10
Group parts by document: true
Query
Filter
Results

AI - Answer Question From Co...
Function Configuration: Function
Memory
combined_question
results
special_instructions
answer

Format[illegible]

docum[illegible]
text_in[illegible]

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio?flowId=[illegible]
Support Chat Bot
Deploy
Publish
Saving...

### Complete chatbot workflow

The chatbot combines the question with conversation history, retrieves relevant information, and passes it to an AI answer node. A final step formats citation references in Markdown.

[1:04](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=64s) [BotDojo canvas showing the chatbot workflow from question input through retrieval, answer generation, and citations.](https://assets.ai.engineer/sites/aiecode2025/sha256/e9065483782a05bb8afdd90b7fa55d453380695b90be4c4ef991cee2cf1a135a.webp)

Conversational AI Assistant
This example implements Retrieval-Augmented Generation (RAG) and then cites the references.
Includes:
• Conversation Memory
• Citations
• RAG Evaluations

Formulate a question from the conversation history.

Answer Question from Context

Format Citation References in Markdown

Start 0
question

Former History
question
history
question

Form new question based on q...
Function Configuration: Function
Memory
history
question
combined_question

Buffer Memory 0
Number of Messages: 10
Memory Key: buffer_memory
User Message
Memory

Special Instructions
Text Output
special_instructions

Retriever 0
Index: [illegible]
Number of Results: 10
Group parts by document: true
Query
Filter
Results

Filter Index
Text Output
Index Filter

AI - Answer Question From Co...
Function Configuration: Function
Memory
combined_question
results
special_instructions
answer

Get Citations
documents
text_input
updated_markdown

End
text

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio?flowId=[illegible]
Support Chat Bot
Deploy
Publish
No issues

### Chatbot answer begins

The chatbot begins answering a vector-index question as workflow nodes execute. The flow uses conversation history and retrieved information to generate an answer with an AI model.

[1:24](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=84s) [Run panel displays the first steps of an answer about creating a vector index, alongside the executing workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/5bbf62ce0b6f69462824d459c90e483402b3ecb8b84c17544de2f468a1c5f9f3.webp)

how do i create a vector index in botdojo

To create a vector index in BotDojo, follow these steps:

1. Organize Your Content: Structure your content into Markdown files. Each file should include Front Matter metadata, which allows you to incorporate tags, categories, and custom fields to provide structure and context to your data[^4].
2. Include Front Matter Metadata: The Front Matter section at the beginning of each Markdown file should contain relevant metadata. This metadata helps in categorizing and tagging your content, making it easier for BotDojo to index and retrieve the information[^4].
3. Point BotDojo to Your Directory: Once

Conversational AI Assistant
This example implements Retrieval-Augmented Generation (RAG) and then cites the references.
Includes:
• Conversation Memory
• Citations
• RAG Evaluations

Answer Question from Context
Formulate a question from the conversation history.
Format Citation References in Markdown

Start 0
question

Buffer Memory 0
Number of Messages: 10
Memory Key: buffer_memory
User Message
Memory

Special Instructions
Text Output
special_instructions

Former History
question
history
question

Form new question based on q...
Function Configuration: Function
Memory
history
question
combined_question

Filter Index
Text Output
Index Filter

Retriever 0
Index: [illegible]
Number of Results: 10
Group parts by document: true
Query
Filter
Results

AI - Answer Question From Co...
Function Configuration: Function
Memory
combined_question
results
special_instructions
answer

Get Citations
documents
text_input
updated_markdown

End
text

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio/?flowId=3dda31e1-238f-11ef-8af8-9d0960812517&left=run
Support Chat Bot
Deploy
Publish
Run
Open Debug Panel
New Session
View Trace
No issues

### Answer with image and citations

The chatbot returns an answer with an index screenshot and a footnote. Beside it, the workflow uses chat history and retrieved information to answer the question with an AI model.

[1:28](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=88s) [Chatbot response includes an index screenshot and footnote links beside the workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f664fc70562858fdca5881f8ffd62105930abb87aacaa45c2fd409363a1f4db.webp)

4. Automatic Indexing: BotDojo will process the Front Matter and index your content, making it easily accessible for your AI application to leverage. This ensures that your data is well-organized and ready for use in various AI-driven tasks¹.

By following these steps, you can efficiently manage and index your data, enabling your AI application to access and utilize the information effectively. BotDojo's approach to indexing ensures that your content remains structured and easily retrievable, enhancing the overall performance and usability of your AI solutions¹.

Embedded screenshot:
MyIndex
Update Index  Download Index  Test Index  Edit Index
Index: MyIndex  Edit
Documents: 7 documents
Embeddings: text-embedding-3-large
Indexed: 7 documents
Dimensions: 3072
Text Splitter: recursive 2000 / 20
Folder: /myindex/
Index is up to date
File Name  Index Status  Content Type  Modified Date  Created Date
[illegible]

Footnotes
1. How do I index data for my AI application using BotDojo? ↩ ↩2 ↩3 ↩4 ↩5

Workflow:
Conversational AI Assistant
This example implements Retrieval-Augmented Generation (RAG) and then cites the references.
Includes:
• Conversation Memory
• Citations
• RAG Evaluations

Answer Question from Context
Formulate a question from the conversation history.
Format Citation References in Markdown

Start 0
question

Buffer Memory 0
Number of Messages: 10
Memory Key: buffer_memory
User Message
Memory

Special Instructions
Text Output
special_instructions

Former History
question
history
question

Form new question based on q...
Function Configuration: Function
Memory
history
question
combined_question

Filter Index
Text Output
Index Filter

Retriever 0
Index: [illegible]
Number of Results: 10
Group parts by document: true
Query
Filter
Results

AI - Answer Question From Co...
Function Configuration: Function
Memory
combined_question
results
special_instructions
answer

Get Citations
documents
text_input
updated_markdown

End
text

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio/?flowId=3dda31e1-238f-11ef-8af8-9d0960812517&left=run
Support Chat Bot
Deploy  Publish
Run
Open Debug Panel  New Session
Checking Answer  View Trace
Type your question...
No issues

### Language model trace

The debug trace shows GPT-4o settings and the system and user prompts, including retrieved document context. The chat response includes an image and citations.

[2:03](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=123s) [A chatbot response beside a debug panel showing model settings and prompt context.](https://assets.ai.engineer/sites/aiecode2025/sha256/60c7a12ec67ca87e11e9f3e2ce5bdba29055183ca3aa2fedb351ffc71d1c45c5.webp)

4. Automatic Indexing: BotDojo will process the Front Matter and index your content, making it easily accessible for your AI application to leverage. This ensures that your data is well-organized and ready for use in various AI-driven tasks¹.

By following these steps, you can efficiently manage and index your data, enabling your AI application to access and utilize the information effectively. BotDojo's approach to indexing ensures that your content remains structured and easily retrievable, enhancing the overall performance and usability of your AI solutions¹.

Embedded image:
MyIndex
Update Index | Download Index | Test Index | Edit Index
Index: MyIndex
Documents: 7 documents
Embeddings: text-embedding-3-large
Indexed: 7 documents
Dimentions: 3072
Text Splitter: recursive 2000 / 20
Folder: /myindex/
Index is up to date

File Name | Index Status | Content Type | Modified Date | Created Date
Fruit cake.txt | Up to Date | text/plain | [illegible] | [illegible]
Fred Meyer.txt | Up to Date | text/plain | [illegible] | [illegible]
Ford Model T.txt | Up to Date | text/plain | [illegible] | [illegible]
Folsom State Prison.txt | Up to Date | text/plain | [illegible] | [illegible]

Footnotes
1. How do I index data for my AI application using BotDojo? ↩ ↩2 ↩3 ↩4 ↩5

Conversational AI Assistant
This example implements Retrieval-Augmented Generation (RAG) and then cites the references.
Includes:
• Conversation Memory
• Citations
• RAG Evaluations

Trace:
request
Former History
Form new question based on question
Form new question based on question
Form new question based on question
Filter Index
Retriever 0
Special Instructions
AI - Answer Question From Content
AI - Answer Question From Content
AI - Answer Question From Content
Get Citations
Session State
correctness_with_score

Time: 4 s 631 ms
Type: gpt-4o
Schema: null
options:
{
  "modelId": "gpt-4o",
  "temperature": 0.04
}
system:
Answer the question based only on the contexts given in the user's query.
Avoid statements like 'Based on the context, ...' or 'The context information ...' or anything along those lines.
user:
Let's think step by step.
1. Consider the following:
contexts:
```
document name: /My BotDojo Docs/docs.botdojo.com/docs/learn/concepts/vector/creating-index.md
document reference: [^0]
# Creating a Vector Index | BotDojo**Why would you create multiple indexes on the same data?**Creating multiple indexes on the same dataset can be crucial for optimizing performance for specific use cases. Each p...
Show More

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio/?[illegible]
Support Chat Bot
Deploy
Publish
Run
Close Debug Panel
New Session
Show Raw Logs
Check Passed
View Trace
Type your question...
No issues

### Batch runs

BotDojo Batches runs sets of questions through a chatbot or AI flow for evaluation. The two listed runs each processed 23 requests with no errors.

[2:28](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=148s) [A batches table showing two completed chatbot runs, each with 23 successful requests.](https://assets.ai.engineer/sites/aiecode2025/sha256/83880f8e06005be8bd7f21961fc870c123f1c3bba25262da0acf506c9565dee3.webp)

Batches

Archive Batch
Compare Batches
Clone Batch
Create Batch

Name | Status | Flow | Data Set | Total Processed | Total Errors | Total Success | Modified Date | Cre[illegible]
Use this is something breaks :( | ✓ | Support Chat Bot | Test Questions 23 | 23 | 0 | 23 | 6/26/2024, 3:28:53 PM | 6/2[illegible]
Without Generated Data | ✓ | Support Chat Bot | Test Questions 23 | 23 | 0 | 23 | 6/23/2024, 5:17:04 AM | 6/2[illegible]

Rows per page 50
1-2 of 2

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio/?flowId=[illegible]&selectedNodeId=botdojo/node/retriever/botdojo_0&sessionId=[illegible]&left=run_batc...
Support Chat Bot
Deploy
Publish
End
https://app.botdojo.com/awesomesauce/projects/my-project/batches/[illegible]...

### Batch evaluation summary

The completed batch processed 23 requests with no errors. Across five evaluations, the green share ranges from 65% for correctness to 87% for groundedness.

[2:32](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=152s) [A completed batch summary with five evaluation bars and 23 successful requests.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f4c26b33c1ee3258a32260970156064e5ff63b552ea1f032b58b7498124a6ea.webp)

Batch - Without Generated Data

Completed
View Log
Edit Batch
Clone Batch

Summary
Requests

Batch Summary
Dataset: Test Questions 23
Created: 6/21/2024, 12:44:01 PM
Flow: Support Chat Bot
Success: 23
Processed: 23
Errors: 0
Duration: 0h 1m 0s

Evaluation Summary
Enough Information: 78%, 22%
Correctness: 65%, 35%
Groundedness: 87%, 13%
Answered Query: 70%, 30%
Answer Relevancy: 74%, 26%
X-axis ticks: 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100

Interface text:
Bot
[illegible]otdojo.com/awesomesauce/projects/my-project/flows/studio/?[illegible]

### Configure a batch run

The presenter clones the batch, raises concurrency to 29, and removes the filter so the run can use generated data. The review screen also shows model and evaluation settings.

[2:55](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=175s) [The review step for a batch run, showing concurrency, evaluation options, index and language model settings.](https://assets.ai.engineer/sites/aiecode2025/sha256/688e38b5dd67e7c304ce6d5d2d97bdecaf04c9caddab67fbefe09261df94efd8.webp)

Create Batch

1 Select Flow and DataSet
2 Map to Flow Inputs
3 Review and Run

Run Options
Name: With generated data
optional
Tags: Add a tag
Concurrency - 29
Adjust how many requests run simultaneously. Increase cautiously to avoid throttling issues.

Evaluation Options
Correctness
Enough Information
Groundedness
Answered Query
10 of 5 evals selected and will run for each flow request
Wait for all evals to complete

Support Chat Bot Properties
Index: My BotDojo Docs

Language Model
Provider: OpenAI
Model: GPT-4o
Temperature

Index Filter
1

Interface text:
Back
Create And Run

### Generated document flow and traces

The presenter opens the flow used to generate synthetic data. It retrieves reference documents, generates a document and evaluates it; the traces table shows runs and evaluation results. The canvas calls for human review.

[3:35](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=215s) [A visual flow for retrieving reference documents, generating a document and running evaluations, with a human-review warning.](https://assets.ai.engineer/sites/aiecode2025/sha256/e49e1d0c7f992d9dc28225c13dcb10282b1df98d361e1b4d8cb79c37ea9cd57f.webp)

Generate Documents from Extracted Q&A

[illegible] documentation.

DEMO ONLY - !!!! ZERO SLOP ZONE !!! - HUMAN REVIEW NEEDED !!!
Don't be a part of the problem

Step 1: Get reference documents
Step 2: Generate Document
Step3: Run Evaluations on Document
Step4: Add to Vect[illegible]
passed)

Start 0
answer
question
questionId

Query Index
Index: [illegible]
Number of Results: 3
Group parts by document: false
Query
Filter
Results

AI - Generate Content
Function Configuration: Function
Memory
answer
question
results
generated_content

Format Result for Evaluation
generated_content
Results

Enough Information 0
Provider: OpenAI
Model: GPT-4 Turbo Preview
Temperature: 0.7
Skip Evaluations: false
context
question
classification
error
meta
passed
reason
score

Vector Index
Index: [illegible]
Index Instan[illegible]

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio?flowId=[illegible]
Deploy
Publish
No issues
Processing 23 items of 23
Dismiss

[3:40](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=220s) [A traces table lists document generation runs and evaluation results.](https://assets.ai.engineer/sites/aiecode2025/sha256/c13fbeabec0b4078a18dfe2fb3190a9f634373112bbea094192593847d739546.webp)

Generate Documents from Extracted Q&A

Traces
Add To DataSet
Add To Feedback Queue
Run Evals

Columns: Trace; Duration; Status; body.answer; body.question; body.questionId; response.passed_evals; response.generated_content; response.generate[illegible]

Visible rows all show a green status check, a checked response.passed_evals box, body.answer “This issue likely occurs becau...”, body.question “When a make a change to a fl...”, body.questionId “slack-releases”, and a final value beginning “generated/slack-re...”.

Trace; 9.23s; ### # Changes to Flow Not R...
Trace; 10.84s; ### # Changes to Flow Not R...
Trace; 28.46s; ### # Changes to Flow Not R...
Trace; 17.22s; ### # Resolving Flow Chang...
Trace; 13.16s; ### # Resolving Flow Chang...
Trace; 10.30s; Here's the document I've crea...
Trace; 11.11s; ### # Resolving Flow Chang...
Trace; 12.35s; ### # Resolving Flow Chang...
Trace; 9.48s; ### # Resolving Flow Chang...
Trace; 13.77s; ### # Resolving Flow Chang...
Trace; 35.80s; ### # Resolving Flow Chang...

Rows per page 50
1-50 of 247

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio?flowId=[illegible]&left=request
Deploy
Publish
Run Flow
Processing 23 items of 23
Dismiss

### Run the document generation flow

The presenter pastes JSON from a previous run containing a support question and answer. The flow uses reference documents to generate a document and run evaluations.

[3:43](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=223s) [A JSON input is shown beside the document generation and evaluation flow.](https://assets.ai.engineer/sites/aiecode2025/sha256/16b7d3690c56afe2d12e4b509afb30f312ef78931329f9d944521f75d0bf2ed8.webp)

Generate Documents from Extracted Q&A

[illegible] relevant information from the existing
documentation.

DEMO ONLY - !!!! ZERO SLOP ZONE !!! - HUMAN REVIEW NEEDED !!!
Don't be a part of the problem

[illegible]nce documents
Step 2: Generate Document
Step3: Run Evaluations on Document
Step4: Add to Vect[illegible]
passed)

[illegible]y Index
[illegible]lts: 3
[illegible]document: false
Results

AI - Generate Content
Function Configuration: Function
Memory
answer
question
results
generated_content

Format Result for Evaluation
generated_content
Results

Enough Information 0
Provider: OpenAI
Model: GPT-4 Turbo Preview
Temperature: 0.7
Skip Evaluations: false
context
question
classification
error
meta
passed
reason
score

Vector Index
Index: [illegible]
Index Inst[illegible]

Run
Open Debug Panel
New Session
JSON
{
  "answer": "This issue likely occurs because Slack is linked to a spec[illegible]
  "header": {},
  "stream": "none",
  "channel": "api",
  "question": "When a make a change to a flow I can see the changes in[illegible]
  "questionId": "slack-releases"
}
Generate Sample Input
Send

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio/?flowId=[illegible]&left=run
Deploy
Publish
No issues
Processed 1 items of 23
Dismiss

### Generated document result

The flow uses a support question and answer with relevant indexed information to draft a document. The example explains how to publish, release, and verify a flow update in Slack.

[4:01](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=241s) [A completed support document explains how to publish and verify a flow update.](https://assets.ai.engineer/sites/aiecode2025/sha256/15c6868fac1fe87f47116a1b0db61d9620fef918043283d1abd81a07d1a3bcd0.webp)

Changes to Flow Not Reflected in Slack

If you've made changes to a flow in the BotDojo Designer but don't see these changes reflected in Slack, this issue is likely due to Slack being linked to a specific release of your flow, often the default one. Integrations like Slack, Teams, API, and embedding chat widgets can all be tied to a particular release.

When you don't have any releases, the integration will always point to the latest version. However, once you have a release, it will use the default release if not specified otherwise.

To ensure that your latest changes are reflected in Slack, follow these steps:

1. Publish Your Changes:
   - Click on the "Publish" icon in the Flow Designer.
2. Create a New Release:
   - After publishing, create a new release of your flow. This will ensure that the latest version of your flow is available for integration with Slack.
3. Verify the Update:
   - Check if the new release resolves the issue by interacting with your bot in Slack.

By following these steps, you can ensure that the changes made in the Flow Designer are properly reflected in your Slack integration. This process allows you to have one version of your flow running in production while you continue to iterate and make changes in the Flow Designer.

Synthetic Data Generation
Generate a document based on the question and answer. Including relevant information from the existing documentation.

DEMO ONLY - !!!! ZERO SLOP ZONE !!! - HUMAN REVIEW NEEDED !!!
Don't be a part of the problem

[illegible] reference documents
Step 2: Generate Document
Step3: Run Evaluations on Document
Step4: Add to Vector Index (if eval passed)

Query Index
[illegible]er of Results: 3
[illegible] parts by document: false
Results

AI - Generate Content
Function Configuration: Function
Memory
answer
question
results
generated_content

Format Result for Evaluation
generated_content
Results

Enough Information 0
Provider: OpenAI
Model: GPT-4 Turbo Preview
Temperature: 0.7
Skip Evaluations: false
context
question
classification
error
meta
passed
reason
score

Vector Index
Index: [illegible]
Index Instance

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio/?flowId=[illegible]&left=run
Generate Documents from Extracted Q&A
Deploy
Publish
Run
Open Debug Panel
New Session
View Trace
No issues
Completed
Dismiss

### Evaluate and index the document

The flow checks whether the generated document contains enough information to answer the question, then sends it to the vector index if the evaluation passes.

[4:19](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=259s) [The completed document appears beside evaluation, vector-index, and output nodes.](https://assets.ai.engineer/sites/aiecode2025/sha256/58dcbb38ec152d647656e1e1643a6cd97e91e4e01a2fe28db6f3553d0c431661.webp)

Generate Documents from Extracted Q&A

Changes to Flow Not Reflected in Slack

If you've made changes to a flow in the BotDojo Designer but don't see these changes reflected in Slack, this issue is likely due to Slack being linked to a specific release of your flow, often the default one. Integrations like Slack, Teams, API, and embedding chat widgets can all be tied to a particular release.

When you don't have any releases, the integration will always point to the latest version. However, once you have a release, it will use the default release if not specified otherwise.

To ensure that your latest changes are reflected in Slack, follow these steps:

1. Publish Your Changes:
   • Click on the "Publish" icon in the Flow Designer.
2. Create a New Release:
   • After publishing, create a new release of your flow. This will ensure that the latest version of your flow is available for integration with Slack.
3. Verify the Update:
   • Check if the new release resolves the issue by interacting with your bot in Slack.

By following these steps, you can ensure that the changes made in the Flow Designer are properly reflected in your Slack integration. This process allows you to have one version of your flow running in production while you continue to iterate and make changes in the Flow Designer.

DEMO ONLY - !!!! ZERO SLOP ZONE !!! - HUMAN REVIEW NEEDED !!!
Don't be a part of the problem

Step3: Run Evaluations on Document
Step4: Add to Vector Index (if eval passed)

Format Result for Evaluation
generated_content
Results

Enough Information 0
Provider: OpenAI
Model: GPT-4 Turbo Preview
Temperature: 0.7
Skip Evaluations: false
context
question
classification
error
meta
passed
reason
score

Vector Index
Index: [illegible]
Index Instance

Code - Write to Vector Index
answer
generated_content
index
passed_contains_info
question
question_id
results
generated_content
generated_filename
passed_evals

End
generated_content
generated_filename
passed_evals

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio/?flowId=[illegible]&left=run
Run
Open Debug Panel
New Session
Deploy
Publish
View Trace
No issues
Completed
Dismiss

### Write evaluated content to the index

The TypeScript node writes the generated document to the vector index when it passes the information check, then returns the filename, content, and evaluation status.

[4:25](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=265s) [A code panel shows a conditional document write and returned evaluation status.](https://assets.ai.engineer/sites/aiecode2025/sha256/15fa7e7af0c8c91c4808d33767732cd011c5a5834e93ddad2693ff60519cd742.webp)

Changes to Flow Not Reflected in Slack

If you've made changes to a flow in the BotDojo Designer but don't see these changes reflected in Slack, this issue is likely due to Slack being linked to a specific release of your flow, often the default one. Integrations like Slack, Teams, API, and embedding chat widgets can all be tied to a particular release.

When you don't have any releases, the integration will always point to the latest version. However, once you have a release, it will use the default release if not specified otherwise.

To ensure that your latest changes are reflected in Slack, follow these steps:

1. Publish Your Changes:
   - Click on the "Publish" icon in the Flow Designer.
2. Create a New Release:
   - After publishing, create a new release of your flow. This will ensure that the latest version of your flow is available for integration with Slack.
3. Verify the Update:
   - Check if the new release resolves the issue by interacting with your bot in Slack.

By following these steps, you can ensure that the changes made in the Flow Designer are properly reflected in your Slack integration. This process allows you to have one version of your flow running in production while you continue to iterate and make changes in the Flow Designer.

DEMO ONLY - !!!! ZERO SLOP ZONE !!! - HUMAN REVIEW NEEDE[illegible]
Don't be a part of the problem

Step3: Run Evaluations on Document
Format Result for Evaluation

Code - Write to Vector Index
Code | Input Schema | Output Schema | Runtime Options
code.ts

async function execute(
    input: InputSchema,
    node: INode,
    context: IFlowContext): Promise<OutputSchema> {

    if (input.passed_contains_info ) {
        let filename = "generated/" + input.question_id + '.md'
        await input.index.addTextDocument(filename, input.generated_content, {
            question: input.question,
            answer: input.answer
        });

        return {
            generated_filename: filename,
            generated_content: input.generated_content,
            passed_evals: true
        }
    }
    return {
        generated_filename: null,
        generated_content: null,
        passed_evals: false
    }
}

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio/?flowId=ba238310-2375-11ef-bafe-71702cc78bfb&selectedNodeId=botdojo/node/code_node_a763f&left=run&right=node_detail
Generate Documents from Extracted Q&A
Deploy | Publish
Run | Open Debug Panel | New Session
View Trace
No issues
Completed | Dismiss

### Selecting batches to compare

The presenter selects runs with and without generated data for comparison. Each shows 23 processed, 23 successful, and zero errors.

[4:54](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=294s) [The batch list shows two selected runs, with 23 successful items in each.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d86d0016924c494d33530933cfb17eb7273f280f2441cdbf41bc5b7dc355a27.webp)

Batches

2 of 3 row(s) selected
Clear Selection

Name | Status | Flow | Data Set | Total Processed | Total Errors | Total Success | Modified Date
With generated data | ✓ | Support Chat Bot | Test Questions 23 | 23 | 0 | 23 | 6/26/2024, 4:06:54 PM
Use this [illegible] breaks :( | ✓ | Support Chat Bot | Test Questions 23 | 23 | 0 | 23 | 6/26/2024, 3:28:53 PM
Without Generated Data | ✓ | Support Chat Bot | Test Questions 23 | 23 | 0 | 23 | 6/23/2024, 5:17:04 AM

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio/?flowId=[illegible]&left=run_batch
Support Chat Bot
Deploy
Publish
Archive Batch
Compare Batches
Clone Batch
Create Batch
Rows per page 50
1-3 of 3
AIE
AI Engineer World's Fair
Presented by Microsoft
Microsoft
smol ai

### Batch comparison error

The batch comparison dialog remains on a loading spinner as the demo reports an error loading the summary.

[4:57](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=297s) [The batch comparison dialog is loading while an error reports that the summary could not be loaded.](https://assets.ai.engineer/sites/aiecode2025/sha256/ae83073525d3b67aba2a9da7babfe8118542a4c91a4228d22853219a8d3c8a16.webp)

Compare Batches
Summary
Data

Error loading batch summary

Interface text:
Flows - BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio/?flowId=[illegible]&left=run_batch
Support Chat Bot
Deploy
Publish
Batches
2 of 3 row(s) selected
Rows per page 50
1-3 of 3
AIE
AI Engineer World's Fair
Presented by Microsoft
Google Cloud
DATASTAX
Lambda
Fireworks AI

### Batch evaluation comparison

The comparison shows more passes with generated data across Enough Information, Correctness, and Answered Query. The presenter calls the improvement measurable.

[5:16](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=316s) [Pass and fail bars compare runs with and without generated data across evaluation measures, including correctness and answered query.](https://assets.ai.engineer/sites/aiecode2025/sha256/1510c06bf748640f3fe9ba1bc1264491e123cb6574f1894ef93409df496ef754.webp)

Compare Batches
Summary | Data

Pass | Fail
Axis ticks: 0 10 20 30 40 50 60 70 80 90 100

Enough Information
With generated data
Without Generated Data
Axis ticks: 0 10 20 30 40 50 60 70 80 90 100
Pass | Fail

Correctness
With generated data
Without Generated Data
Axis ticks: 0 10 20 30 40 50 60 70 80 90 100
Pass | Fail

Answered Query
[illegible] generated data
[illegible] Generated Data
Axis ticks: 0 10 20 30 40 50 60 70 80 90 100

Interface text:
BotDojo
app.botdojo.com/awesomesauce/projects/my-project/flows/studio?flowId=[illegible]&left=run_batch
Support Chat Bot
Deploy
Publish
Batches
2 of 3 row(s)
Rows per page: 50
1-3 of 3
AIE
AI Engineer World's Fair
Presented by Microsoft
Google Cloud
DataStax
Lambda
Fireworks AI

## Full transcript

[0:00](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=0s) [on-hold music] So hello, my name is Paul Henry.

[0:15](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=15s) I'm the founder of BotDojo, and as a previous CTO, I was working with teams deploying LLMs applications for hundreds of thousands of customers. And like many of you guys know, it's super easy to hook up a vector database, um, with an LLM over the weekend, but really hard to get it pr-production ready.

[0:33](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=33s) And so that's what we do. We are an AI enablement company, and we let companies deploy AI to prod.

[0:41](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=41s) Live demo time. All right, so today I'm going to show you a, a demo of our product. We're going to take, uh, synthetic data that we're gonna generate, and we're gonna combine it with e-evaluations to see how we can improve the performance of a chatbot.

[0:56](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=56s) Or at least that's what I hope happens. All right. So I'm gonna open up our template of our, uh, a chatbot, and we have customers live that are using this template.

[1:07](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=67s) It's kind of battle tested. Um, and so let's test it out.

[1:14](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=74s) How do I create a vector index in BotDojo?

[1:23](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=83s) Okay. And as you can see, all the little nodes are lighting up as they execute. Um, we're taking the question, we're looking at the chat history, we're going to the ve-vector database to retrieve the information, and then we're answering it with a AI model.

[1:36](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=96s) So if I pull this up, you can kind of see in our low-code, uh, editor, this is the prompt that we're sending to the LLM. We're getting the results out here.

[1:45](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=105s) And we also support, uh, JA- JSON Schema. So if the model, uh, supports JSON output like, um, Groq, um, Claude and all that stuff, then we just conform to that.

[1:57](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=117s) Um, one key thing is you can pull a trace of each node and see exactly what we sent to the LLM, what came from the retriever, the exact, you know, data, which has been super useful for debugging apps.

[2:11](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=131s) All right. And cool, we have an image. It's got citations. We should ship it.

[2:17](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=137s) That was supposed to be a joke, but all right. Um, [laughs]

[2:21](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=141s) so this is where evaluations come in. So I'm going to demonstrate, um, the evaluations that I previously ran. So we have a, a feature in BotDojo, uh, called Batches, which allow you to run a whole bunch of questions through your chatbot or your AI flow and, um, run evaluations to kind of see how things are doing.

[2:40](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=160s) So if you can see this, we have a few, uh, five evaluations that we ran. There's a little bit of red. Um, that's because, uh, we don't have enough information from our vector database.

[2:49](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=169s) Um, it also checks for things like hallucinations. So let's try to fix that. And so I'm going to clone this batch. I'm gonna rename it with generated data. I'm gonna increase the throughput a little bit because of time.

[3:06](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=186s) And, um, I had... I don't have enough time to generate all the data for this demo, so, um, the previous run was filtering out the generated data, and so I'm gonna remove the filter that we're passing into the, uh, flow so it, it takes in the generated data.

[3:21](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=201s) You can also change the model and all that kind of stuff to see how it performs.

[3:26](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=206s) All right. So while that guy is running, I'm going to open up another flow. And so this is the actual flow that we, uh, generated that, uh, synthetic data.

[3:37](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=217s) And so let me, uh, let me run this one real quick.

[3:43](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=223s) And so this particular flow takes in multiple inputs, and so I'm gonna paste in, uh, some JSON from a previous run.

[3:52](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=232s) And what this is gonna do is, is kind of a trick that's been working well for customers, is where you take, um, you extract questions and answers from support tickets, so these are live agents talking with customers, and you use this as a test data to send it through your chatbot.

[4:07](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=247s) And, um, we take relevant information from the existing index, and we have it write a document. Um, and so it, it uses the same writing style and it, um, you know...

[4:18](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=258s) And then we do a inline cite ta- uh, evaluation to where we check to see if the document has enough information to answer the question. And then we also have a code node here where, you know, a lot of times when you're using these low-code editors, there's like situations where you have forty thousand different, um, boxes.

[4:34](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=274s) And so when you have to do write code, we support, um, TypeScript and, um, and soon, uh, Python. But you can see that, hey, we're getting the information and we're writing to the vector index.

[4:45](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=285s) All right. Running out of time. Okay. Let me go back to the support chatbot. Yeah, moment of truth. So I'm gonna compare, um, the, the batch that we ran before with the new stuff in twenty seconds.

[5:01](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=301s) Oh, sh- [laughs] You do it, you do it fifteen times, and it doesn't work.

[5:10](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=310s) Ten, nine... We're also hiring, so if you're an AI engineer- [laughs] [laughs]

[5:14](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=314s) Help, help us fix this. All right, there it goes. Okay. Phew. All right. One second left. It's all green, so it improved the, uh, you know, [laughs] measurably improved something.

[5:23](https://www.youtube.com/watch?v=PNjMwdCo_YM&t=323s) So, uh, thank you. Um, BotDojo.com. Check us out. Thanks. [upbeat music]
