Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked

Engineering context lives across pull requests, review comments, tickets, wikis and chat. Those records contain meaning, but they also contain authors, dates, states, labels and relationships. Peter Werry, who builds context engines at Unblocked, uses that distinction to frame the workshop: retrieve the discussion that sounds relevant, and you may still miss the records the question actually asks for.

Here, “RAG” has a deliberately narrow meaning: semantic similarity search or vector retrieval. Embed the question, find documents with similar meaning, rerank them and put the selected content into a model’s prompt. That works well as a starting point for finding code, a PR that changed it or a Slack thread that debated it. An agent can then inspect files and follow leads through GitHub.

“PRs I merged last week” exposes the missing mechanism. Nearest-vector search can find text about merging or about last week without enforcing who performed the action, whether the PR was merged or when that happened. A semantically convincing result could have been merged three years ago. Reranking for relevance does not by itself enforce those structural and temporal conditions.

The other half of retrieval is a database query that expresses those conditions. The model’s job changes: it writes the query that accesses the records. The database supplies the matching data. This separation is the foundation for the six stacked pull requests in the workshop, which progressively replace a stubbed engine with a working one.

A request such as “Show me Peter’s open PRs” looks simple until it reaches the database. Peter may not be the value stored in the author field. A generated query may also use an unsafe operator, or fail even when its intent is sensible. Four supporting mechanisms make query generation useful:

• Schema awareness. Give the model the fields and types it can query.
• Identity resolution. Map a person’s name to the identifier stored in the records.
• Validation. Treat the generated query as untrusted output and restrict what it can execute.
• Retries. Return validation and execution errors to the model so it can revise its plan.

The starting application already has a query playground and a chat tab, but the backing query engine is a stub. Ingestion loads repository content first; the workshop then builds around the existing entry point. Keeping the interface in place makes the changes observable: the playground begins unable to answer anything and becomes usable when query planning is connected in step three.

The ingestion script knows enough about its records to download and store them. A separate query service may not have that knowledge. Unlike reading table definitions from a relational database, this MongoDB example reconstructs a description from the documents themselves.

Schema discovery samples 300 documents and merges their nodes into one tree. Null values are skipped because they do not constrain the type of a field. Procedural rules infer types and recognize enumeration patterns. An LLM could perform extraction, but its variable output would still need structural checks; this implementation puts the extraction itself in ordinary code.

What information will the planner actually receive? The discovery run produces a Markdown description of the GitHub pull-request schema and also creates some indexes. Markdown is an intentional output format: the resulting field description becomes part of the prompt used to synthesize queries. Discovery therefore turns sampled document structure into an input the model can consult, rather than asking it to guess the database’s shape.

Knowing that an author field exists does not tell the model which value belongs there. A literal match for Peter can return nothing when GitHub stores a login instead. Identity resolution supplies that missing connection between the person named in the request and the identifier used by the data.

The workshop uses simple fuzzy matching; Werry names Levenshtein distance as a suitable technique. This approach couples the resolver to some knowledge of the schema. Dynamically discovering the mappings with an LLM is another possibility, but the example chooses a smaller procedural solution. The resolver alone does not yet make the playground work: it must be connected to the planning step that follows.

Query planning is the LLM portion of the engine. The prompt receives the schema, and a required tool call performs name extraction as part of producing the plan. Temporal requests need another input: the current date and time. Without that reference, a phrase such as “last week” lacks the anchor needed to generate a useful date condition.

How does a human request become a database condition? With “Open PRs by Peter,” the connected engine generates a match clause and resolves Peter to pwerry, Werry’s GitHub login. The important change is the value used for matching: the request can use a familiar name while the query uses the identity present in the records.

The first working query is a milestone, not permission to execute whatever the model produces. MongoDB’s where or function clauses can run JavaScript on the server. Passing generated queries through unchecked would make the model’s output an execution path for unwanted behavior. Internal ID or metadata fields create a separate risk: the query can expose information the model should not control.

Validation uses “deny by default.” An allowlist defines the permitted operators, so an unfamiliar operator does not become executable merely because nobody remembered to prohibit it. Fields can also be allowlisted, although this dynamically discovered schema example does not implement a general field allowlist. Known internal metadata still needs explicit handling.

Tenant isolation makes the distinction concrete. Werry recommends separate collections for separate tenants. If a collection does contain multiple tenants, hiding its tenant-ID metadata from the model is only part of the job. Application code must also remove any model-supplied version of that metadata and inject the application’s own tenant constraint. The trusted tenant selection comes from code outside the model, and this sanitization should be procedural.

The next demonstration searches for authentication in a PR title. Regular expressions are disallowed, so the validator rejects the generated query. At this stage the error is simply returned. The engine has successfully enforced a restriction, but it still needs a way to recover from the rejection.

The retry loop handles two different failure points. Validation can reject an operator before execution. Execution can reject a query that passed the validator—for example, a projection that improperly mixes inclusion and exclusion. In either case, the engine returns the error to the model and asks for a revised plan, up to a maximum number of attempts.

Where does a failed plan go, and what is allowed to reach the database? The branches below separate validation failure from execution failure. Both can send diagnostic information back to planning; only a validated query proceeds to execution. A retry therefore gives the planner another chance without removing the checks that rejected its earlier output.

Rerunning the authentication query reveals a subtler problem. By the third attempt, the planner has worked around the unavailable regular expressions and full-text search by using an exact match. That query tries to match authentication exactly and fails to find the intended content. What changed across the attempts? The query became acceptable to the available operators, but the remaining operation still did not express the search the user needed.

The correction is to add full-text search and create its index. The engine can then combine text search with author and state filters in the same match. The demonstration succeeds after that addition. This is a useful limit on self-correction: retries can help choose among available operations, but they cannot supply a search capability the database interface does not offer.

The final application step moves to the chat tab and asks, “Did we work on anything authentication related last week?” The agent generates a query, retrieves the data and reasons over it. The query playground and chat agent share the same backing engine; the chat interface adds reasoning over the retrieved records.

The resulting engine supports filtering, aggregations, time-scoped queries and full-text search, with validation and error-driven retries. Werry singles out the error loop as especially important, but also recommends strengthening the validation harness before internal deployment. The workshop establishes an end-to-end implementation; it does not make the sample validator a complete deployment safeguard.

The first audience question widens the example from one pull-request collection to many collections and tables. Putting every schema into every prompt would enlarge the planner’s context. The proposed extra LLM step works with small collection summaries and descriptions, selects the relevant collection, and loads the corresponding schema for the query.

Keeping the records current is a different problem. Unblocked continuously ingests changes: a PR event triggers a webhook, the content is ingested and re-embedded, and the result is stored in the database. Schema selection reduces what the planner must read; the ingestion pipeline keeps its searchable data current. The answer describes that architecture without quantifying processing latency or cost.

Choosing sources depends on the questions people want answered. Two sources carry especially useful decision history:

• Slack conversations. Werry prioritizes Slack for “why” questions. A recollection such as a conversation with a particular person roughly three weeks ago already contains participant and time constraints that structured retrieval can use.
• PR comments. Review discussions hold organizational history that is otherwise easy to lose. Unblocked processes past PR comments into “best practice memories,” distilling useful lessons before serving them as context.

Could a restricted MCP interface replace generated database queries? MCP can expose constrained operations, and its tools can be updated when the schema changes. It can also accept a natural-language request and run the workshop’s translation engine behind the call. The interface choice determines what the client may ask for; the query engine explains how the backend can fulfill that request.

The next question returns to the opening claim: should structured retrieval replace similarity search? “They’re absolutely complementary.” Full-text search finds keywords, while semantic search can match an idea expressed in different words. One useful sequence starts with vector retrieval, gives the agent relevant starting data, and then lets it form structured queries to fetch additional records.

A schema-discovery refresher also narrows what the sample algorithm guarantees. It merges JSON-like document structures and uses token and pattern heuristics to infer types and enumerations. Werry calls this iteration “pretty hacky,” describing a character-count check between two and eight as one heuristic. Procedural extraction makes the rules inspectable; it does not make those rules universally correct.

“Hybrid search” needs its own definition. In the later exchange, it means running semantic and lexical search and combining their rankings through rank fusion. An attendee describes full-text and semantic stages in an Atlas aggregation pipeline, and Werry confirms that Unblocked uses this kind of combination. That ranking combination serves a different purpose from enforcing author, state and date constraints: both can contribute to retrieval.

Why sample 300 documents? The number came from empirical trial and error, including customer reports that retrieval had missed something they expected to find. It is not a completeness guarantee. Document variety affects the sample size a use case needs, and rare shapes can still be missed. Unblocked uses many static schemas to prioritize correctness; dynamic discovery is useful for exploring unfamiliar data and connecting arbitrary databases.

The discussion also clarifies what the required tool call does: schema discovery happens at startup; the enforced call is for name extraction. Two enforcement techniques are described:

• Validate the query tool’s username argument. Make the username an input to the query tool and reject an incorrect or malformed value there.
• Monitor the extraction call. Watch the agent’s actions. If it skips the required extraction tool, insert a message telling it to make that call before continuing.

Finally, schema change requires an explicit maintenance path. An engine discovering an unknown schema can rerun its sampling process periodically as documents arrive. Unblocked’s controlled, static schemas instead change with the codebase: upstream changes are adopted in code and deployed through CI. These choices leave a practical decision for an implementation: use discovery to learn an unfamiliar database, or maintain a schema deliberately when the application controls its shape.