BotDojo Launch: Enhancing AI Assistants with Evaluations and Synthetic Data

Connecting a vector database to an LLM can take a weekend. How do you know the resulting chatbot is ready for customers? Paul Henry, founder of BotDojo and a former CTO, describes that gap from work with teams deploying LLM applications for hundreds of thousands of customers. The connection is easy; making the application dependable is harder.

The demonstration starts with a chatbot template Henry says customers already use. The proposed improvement combines synthetic data with evaluations: identify where the chatbot lacks information, make additional material available to retrieval, and run the questions again to see what changes.

“How do I create a vector index in BotDojo?” is the first live question. As the chatbot runs, nodes light up in the editor: the flow takes the question, consults chat history, retrieves information from the vector database, and sends the resulting context to an AI model to produce an answer. This makes the retrieval-and-generation sequence visible rather than hiding it behind the chat interface.

Opening a node reveals the prompt sent to the LLM and its output. Henry also describes JSON Schema support when the selected model supports JSON output, mentioning Groq and Claude. That presentation-time description is conditional on model capabilities; it does not establish a particular provider API or strict schema-enforcement guarantee.

Per-node traces expose the exact data at each boundary, including what the retriever returned and what the LLM received. Those are different debugging questions: did retrieval supply useful evidence, and did the model use the supplied evidence well? The answer on screen includes an image and citations. Henry jokes, “We should ship it”—setting up the difference between an attractive answer and evidence that the chatbot works across questions.

BotDojo’s Batches feature runs collections of questions through a chatbot or other AI flow and evaluates the outputs. Henry opens an earlier run with five evaluations and attributes its red results to insufficient information in the vector database. The evaluations also check for hallucinations. Five evaluations does not establish how many test questions were in the batch.

To test the additional content, Henry changes the batch setup:

1. Clone the earlier batch and rename it “with generated data.”
2. Increase throughput to fit the live demonstration’s time limit.
3. Remove the index filter that excluded generated content from the earlier run.

The generated documents already exist; there is not enough time to create the full dataset on stage. The substantive retrieval change is to admit previously excluded documents, rather than regenerate the corpus during this rerun.

A small TypeScript example makes that selection difference concrete. Here, generated records are present in the corpus in both cases; only their eligibility for retrieval changes:

type DocumentRecord = {
  id: string;
  generated: boolean;
};

const corpus: DocumentRecord[] = [
  { id: "existing-document", generated: false },
  { id: "support-derived-document", generated: true },
];

const baselineCandidates = corpus.filter(doc => !doc.generated);
const expandedCandidates = [...corpus];

These illustrative records express the filter operation, not BotDojo’s filter syntax. In the demo, removing the filter lets the flow retrieve generated material alongside existing content.

Henry notes that model selection can also be changed for comparisons. He does not demonstrate a model change here; the comparison he sets up concerns access to generated content.

While the batch runs, Henry opens the flow used to generate the synthetic data and starts an example run. It accepts multiple inputs, supplied here by pasting JSON from a previous run. The starting material is support tickets: conversations between live support agents and customers. Extracting their question-answer pairs supplies questions to send through the chatbot as test data.

The document-generation process then connects those support interactions to the existing knowledge base:

1. Retrieve relevant information from the existing index.
2. Use that information to write a document in the same writing style.
3. Evaluate inline whether the document contains enough information to answer the question.

The evaluation asks a specific question about the generated document’s usefulness. Producing plausible prose alone is not the completion criterion being checked.

A code node handles logic that would otherwise require a sprawling collection of low-code boxes. Henry describes TypeScript support and Python as coming soon at presentation time. The flow then writes the generated information into the vector index, making it available to the support chatbot when retrieval permits it. This closes the path from support conversations, through document generation and a sufficiency check, back into the corpus being tested.

Henry returns to the support chatbot to compare the earlier batch with the run that includes generated content, with roughly twenty seconds remaining. The comparison initially fails to appear, prompting a joke about having rehearsed it fifteen times. As the countdown continues, he adds an appeal for AI engineers to join the company and help fix it.

The comparison finally appears with a second to spare. Henry reports that the final comparison is all green, but supplies no numerical improvement, named final metric or test-question count. His closing description—“measurably improved something”—captures both the successful reveal and its limits. The demonstrated loop is concrete: evaluate a chatbot, admit additional support-derived documents into retrieval, and compare the rerun. The recording ends with that comparison, rather than a production deployment walkthrough.