AI on Your Lakehouse: Context Comes in Shapes, Not Queries

An auto-repair copilot can retrieve the right records and still give the wrong answer. Its warehouse contains structured repair history; its document store contains manuals, safety bulletins, and recalls. Text-to-SQL and vector search make those sources accessible, but access alone does not ensure that an agent sees the relationships it needs.

Consider Autofix, the fictional national repair chain used throughout the workshop. A technician has a vehicle in a bay and needs to know how to repair it. Leadership wants a copilot to support that work, and the AI engineer has to connect the technical library to the history of repairs already performed. The demonstration uses cloud-stored documents and BigQuery, with patterns intended to extend to Databricks and Snowflake. As the number of similar tables and documents grows, relevant context can fall through the cracks even when individual retrieval calls work correctly.

The harder questions concern the whole operation: Which incoming vehicles lack documentation? Which documents are never used? Which repairs keep failing? Which recalls form recurring groups? A similarity search returns things resembling a query; establishing a missing connection requires examining coverage and relationships. Even a seemingly ordinary warehouse question can depend on choosing the correct joins among many lookalike tables.

The context an agent receives should fit the question it is answering. The workshop introduces three shapes:

The implementation starts with Connections, then builds document navigation and Themes.

The underlying model is a property graph. Nodes represent things such as people and cars; relationships represent associations such as a person owning or driving a car. Both can carry properties, including strings, numbers, dates, and vectors. Blumenfeld compares the connected structure to pre-joined tables: once relationships exist, a query can follow them instead of reconstructing each association from scratch.

The GraphAcademy workshop organizes the exercises into seventeen lessons. To follow along:

1. Sign in, enroll in the course, and open its Environment lesson.
2. Launch the preconfigured GitHub Codespace in your own account. A local clone is also possible through the repository’s Bash setup script, but requires the local dependencies, including Claude Code.
3. Wait for initialization, then populate the environment file with Anthropic access, BigQuery credentials, and your individually provisioned Neo4j sandbox credentials.
4. Start Claude Code. In the demonstration, opening a fresh terminal resolves the case where it asks for a subscription instead of recognizing the API key. Accept the configured MCP server and ask the agent to check its Neo4j connection.

Blumenfeld’s graph already contains data from an earlier run; a new participant’s sandbox begins empty.

The coding workflow uses the Neo4j CLI to execute database queries, a Cypher skill to guide query construction, and a Graph Data Science skill for the later algorithms. Rather than handwrite every query, participants give the agent specifications for the outputs they want.

The repository separates those concerns: .claude contains skills; outline, search, and themes scripts contain implementation placeholders; a small Python SQL runner calls BigQuery; and Markdown specifications describe the required outputs. Solution scripts provide a recovery path when generated queries fail. The repository also includes data-loading material, Databricks directions for comparisons with Genie and AI Search, source PDFs, and a readable Markdown corpus. Cursor is a possible local alternative, though Blumenfeld has not tested the workshop with it.

The warehouse schema centers on work orders, with vehicles, diagnostic trouble codes (DTCs), procedures, work-order parts, and parts connected through primary and foreign keys. The first task is to make those relationships legible to the agent. NeoCarta, a Neo4j Labs project, supplies a graph representation of warehouse metadata. It can also represent business terminology and processes, although the workshop does not use those extensions.

Blumenfeld distinguishes an ontology, which helps interpret and reason about data, from a semantic layer, which supplies agreed terms for querying it accurately. A separate virtual-graph approach can expose SQL data through a federated Cypher interface; the exercise instead uses an MCP ingestion operation to build the metadata graph. The workshop load is described as producing six tables and five reference join paths. Its hierarchy runs from database to schema to table to columns, with optional representative values.

The first question is concrete: Which vehicle received part IC2042? The agent reads the NeoCarta metadata through MCP, identifies the relevant join path, and uses the Python runner to query BigQuery. The graph guides warehouse execution without replicating the warehouse wholesale. Representative values may accompany metadata, but the repair records remain in BigQuery and SQL retrieves the answer.

During the setup recap, one detail matters for anyone following along: an absent .env, or a workspace containing only .env.example, can mean initialization is still running. The setup script populates files asynchronously. Each participant must use their own sandbox credentials alongside the service credentials. Once the query completes, the agent’s Python SQL calls perform the selections and joins, and the displayed answer groups vehicles by make and vehicle count. The demonstrated join paths come from BigQuery’s primary/foreign-key metadata; NeoCarta can also use query logs or manually supplied joins, terminology, and metrics.

A graph can improve execution and it can improve an agent’s understanding of a schema. Supply-chain shortest paths and variable-length traversals illustrate the first use. Choosing a short join path from hundreds of candidate tables illustrates the second: the final SQL may be simple even when discovering the right relationship is difficult. NeoCarta addresses that discovery problem while preserving existing warehouse access. Neo4j Virtual Graph, described in the recording as a preview capability, moves the graph interface down to query execution through Cypher pushdown. Related context-graph work is mentioned, but is not part of this implementation.

Why not copy the transactional data into Neo4j? The answer depends on what the copy buys:

• Keep records in the warehouse: continuously updated datasets require synchronization, selective copying can require custom ETL, and security policies may prohibit moving sensitive records into another system.
• Import records for graph workloads: large recursive joins, graph algorithms, graph embeddings, and clustering can justify native graph storage. Unstructured documents also benefit when ingestion gives them a useful structure.

The metadata graph itself persists. Starting NeoCarta does not rebuild it: the build-connections script explicitly invokes ingestion, after which later queries reuse the stored metadata. Blumenfeld expects the load to be idempotent but does not rerun it to demonstrate that property.

The Cypher skill is maintained internally and distributed alongside skills for agent memory, cloud infrastructure, and language drivers. Its purpose is to supply current query guidance instead of letting the agent rely on old web examples. Blumenfeld does not describe the exact process used to author its SKILL.md; the dev-container installation file shows where the CLI and skills are installed.

A question about bronze, silver, and gold data layers receives a roadmap answer rather than an additional demonstration. NeoCarta is customer-driven; at the time of the workshop, a Databricks connector is under development, with metric views and document support among the possible extensions being discussed.

The document side contains simulated manuals, bulletins, and recalls, available as PDFs and Markdown. An ABS manual, for example, refers to platform codes and diagnostic troubleshooting material elsewhere in the library. Those references already carry useful structure. The loader reads local files for the exercise, standing in for the cloud-stored collection.

The Table of Contents shape lets the agent navigate that structure. Blumenfeld invokes PageIndex as an analogy for reading a document hierarchy; it is not a workshop dependency. The outline begins with a library, descends through folders, documents, and sections, and includes cross-links to other documents. Every location has a hierarchical URI. An agent can use a URI to retrieve text, inspect a subtree, or follow a reference into another part of the library. Unlike Connections, this shape imports document content into the graph.

A familiar GraphRAG ingestion pipeline extracts entities—vehicles or parts, for example—and connects them to source chunks. Here the first graph is lighter: a deterministic parser preserves containment, section order, and existing named links, resembling the references in an Obsidian vault. Hyperlinks can supply the same kind of cross-document relationship.

For documents with consistent structure and substantial interlinking, this avoids an LLM extraction pass. The loader is designed to be idempotent, and the hierarchy becomes part of each node’s URI, from library through folder and file to section. Inspecting the loaded graph reveals the three relationships the tools will use: containment through HAS, section ordering, and cross-document references through LINKS_TO.

The outline tool starts as a query placeholder inside the Autofix skill’s script. A Markdown specification describes the human-readable shape and optional arguments, and the agent uses the Cypher skill to implement it. The central operation is a variable-length traversal over HAS; the demonstration shows a bound of *0..25, with the script controlling traversal depth. A second query retrieves cross-links.

The containment operation can be expressed directly in Cypher. Given a root URI, this query returns the root and descendants through the demonstrated bound, together with their depth:

MATCH (root {uri: $rootUri})
MATCH path = (root)-[:HAS*0..25]->(node)
RETURN node.uri AS uri,
       length(path) AS depth
ORDER BY depth, uri;

The zero-length path includes the root itself. This is the traversal core; the complete outline also formats the hierarchy and attaches the separately retrieved references.

Without a root URI or depth restriction, the script returns the entire graph. Blumenfeld can do that with the workshop’s small corpus, but discourages it for larger collections. Setting depth to one gives a shallow view; supplying a URI moves the root. The live example starts at the Falcon 2.0 document, then takes a coil-identification URI from a link and uses it to request another outline. The output of one navigation step becomes the input to the next.

To check whether the agent actually uses these tools, inspect its tool-call history, including NeoCarta MCP calls. Asking the agent to describe its steps is useful for teaching, but production verification belongs in execution logs. The ingestion code is similarly inspectable: the corpus parser uses regex for these consistently structured documents. Messier sources may require additional processing, including GLiNER for entity extraction or an LLM; that does not make either a complete document-structure parser. NeoCarta’s MCP endpoint, its CLI, and the skills approach are not inherently tied to a particular model.

A single HAS relationship across all containment levels makes the variable-length query simple. Separate types such as HAS_FOLDER, HAS_DOCUMENT, and HAS_SECTION would require more cases. The same reasoning keeps LINKS_TO general. More specific relationships can express richer distinctions, but hundreds of types become harder for both people and agents to manage. On the warehouse side, semantic metadata serves a related purpose: it reconciles business terminology with the physical schema.

Navigation complements search. A full-text or vector query can locate a starting point; an outline can expose its neighbors and the larger structure around it. Repeated similarity searches may miss a document that matters when investigating an absence or a coverage gap. Explicit traversal provides a different way to inspect the collection, rather than requiring every next step to resemble the original question.

The workshop also leaves some entities implicit. Instead of creating a dedicated part node for every mention, the agent can read a part from a document and use it to query the warehouse. Blumenfeld sees stronger agents making this practical in some cases. A domain such as life sciences may still require explicit molecule and medication ontologies because precise relationships are essential to valid inference. The decision is about the domain’s required semantics, not a universal requirement to extract everything—or nothing—during ingestion.

The next placeholder implements search using Neo4j’s Lucene full-text index, without vector search. The index covers document and section nodes containing text, not folders, and returns matches with scores. Because each match carries a hierarchical URI, the query can apply a subtree restriction after full-text retrieval: keep only nodes whose URI begins with the requested prefix.

Semantic expansion supplies some vocabulary flexibility without an embedding pipeline. If a user describes an engine shuddering, the agent can expand that description to terms such as misfire or rough idle, then combine them with Lucene’s OR. The demonstration also searches for coils only under recalls. The URI encodes enough containment information to scope those results without an additional graph traversal.

The complete callable procedure is db.index.fulltext.queryNodes(). The following Cypher shows the search-then-filter sequence, with the configured index name, Lucene expression, URI scope, and result limit supplied as parameters:

CALL db.index.fulltext.queryNodes($indexName, $lucene)
YIELD node, score
WHERE node.uri STARTS WITH $scopeUri
RETURN node.uri AS uri, score
ORDER BY score DESC
LIMIT $limit;

For the symptom example, $lucene can contain misfire OR "rough idle". For a recall-only coil search, it contains the coil query and $scopeUri identifies the recalls subtree. The prefix condition filters the returned nodes; it does not change the full-text index into a separate index for that folder.

An agent can help create a missing full-text index, but it still needs a sensible indexing contract. Neo4j supports indexing multiple labels and properties, and that flexibility creates choices the agent may misunderstand. Blumenfeld thinks he may have generated the workshop index with an agent, but does not recall that construction step.

Document names matter because an outline exposes names before content. A poor title gives the agent less evidence for deciding where to navigate; search can compensate by looking inside the document. Outdated or drifted content is another problem. Blumenfeld proposes last-updated or authority metadata as possible additions, and emphasizes descriptive link aliases: they explain why a document points elsewhere. LLM enrichment during ingestion can help when that descriptive information is missing.

A deterministic load can rebuild the same document graph from unchanged sources. Incremental updates are possible too, but a document is not isolated: renaming it changes the URI that other documents may reference. Partial synchronization therefore needs to update or clean the affected subtree and account for incoming links, rather than simply replacing one node.

An audience question about selectively applying PageIndex leads to a useful sequencing clarification. There is no mandatory outline-first order. An agent can inspect an outline and then search within a promising area, or search first and then follow the matching document’s links. The optional practice material explores the Cypher details further, though the live workshop moves on to Themes.

Looking only at LINKS_TO relationships reveals another shape. Manuals, recalls, and bulletins refer to one another; a guide may be the shared target of many repair procedures. Zooming out reveals groups of highly interconnected documents. These groups can expose subjects that were never explicitly tagged in the library.

The workshop uses Leiden community detection to assign documents to groups. The aim resembles the global-versus-local distinction in Microsoft GraphRAG, but the input here is existing document structure rather than an LLM-extracted entity graph. Blumenfeld describes Leiden as a successor to Louvain with more efficient behavior. Neo4j Graph Data Science runs it on an in-memory projection of the relevant graph. His discussion of millions or billions of nodes concerns the intended scale of graph analytics, not a measured workshop run; the projection still requires adequate memory.

The agent receives a compact view of each group rather than a graph visualization:

Conductance relates connections across the community boundary to connectivity within the graph, supporting descriptions such as tightly or loosely interlinked. Link labels and document titles make the groups interpretable: the examples expose a BCM/bus cluster and a braking cluster containing rotor pads and hydraulic lines, without generating new theme names.

The implementation again begins with a specification and a placeholder, this time in themes.py. The agent uses the Cypher and GDS skills to build the projection, while the script calls Leiden through the GDS library. Before projecting, it collapses section-to-section references to their owning documents using the URI structure. That produces document-level relationships, so the resulting communities describe how documents group together rather than fragmenting the view across individual sections.

Blumenfeld reports 13 groups at the default gamma=1 and 14 themes at gamma=2 on the simulated workshop corpus. Gamma controls resolution, allowing a more finely divided view; the appropriate setting depends on the corpus. The output continues to use source link labels and document names. Returned URIs let the agent move from a theme into an outline or scoped search, supporting questions about concentrated issues, thin documentation, or where new material belongs.

Leiden supplies group IDs; describing those groups is a separate choice. The workshop displays source metadata, which avoids generated labels but depends on good document titles and link names. Microsoft GraphRAG-style processing can instead add entity extraction and hierarchical LLM summaries, at additional cost and latency, with potentially varying generated text. Source-derived descriptions avoid that generation variability, but unchanged documents alone do not guarantee identical Leiden assignments: reproducibility also depends on randomness controls and execution settings.

For changing data, rerun the projection and retain snapshots of the communities and their descriptions. Those snapshots let an application examine how groups evolve over time. Blumenfeld points to chargeback fraud and anti-money-laundering analysis as established uses of temporal community analysis, where repeated clustering helps reveal changing patterns.

Large theme views can be filtered by community size or conductance, but small communities may still be important. Filtering the view also does not validate the source. The deterministic parser trusts existing references: a wrong link remains a wrong link, and a malformed section may survive ingestion. Agent traversal could help audit such problems, but the workshop does not demonstrate a complete cleaning solution.

Predefined shapes do not have to cover every possible question. The Neo4j CLI can retrieve the schema and execute custom Cypher, letting a coding agent compose a query when it needs to connect information in an unanticipated way. Blumenfeld reports a better text-to-Cypher experience with the CLI and maintained skills, without presenting a measured comparison. The live CLI exercise is abbreviated, but the schema inspection shows the information available to the agent.

The first combined demonstration returns to the technician: for a given VIN and diagnostic code, what has fixed similar vehicles? The operational goal is to reduce the comeback ratio—customers returning because the previous repair did not work. Warehouse history records those outcomes, while manuals, recalls, and bulletins describe the parts and procedures. A prewritten domain skill exposes the outline, search, and themes scripts alongside the SQL runner and NeoCarta MCP access.

The answer is assembled through several operations:

1. Search for the diagnostic code and related symptoms, including misfire or rough idle.
2. Locate the relevant engine manual and inspect its linked procedures.
3. Retrieve the warehouse join paths.
4. Query the vehicle and comparable work orders involving the diagnostic code and part.
5. Combine the repair history with the documentation to recommend replacing the old ignition coil with its revised part.

The graph supplies the document-to-procedure route; the warehouse supplies what happened in previous repairs.

Blumenfeld’s comparison with vector retrieval is about the path through the evidence. A search-based system can answer this question too, but repeatedly searching for each related procedure introduces opportunities to retrieve a misfire passage about the wrong part. Following the manual’s explicit links gives the agent a more constrained route before it returns to the warehouse.

The supervisor’s question is broader: Are documented procedures consistent with field problems, what documentation is missing, and what documentation is not being used? This requires comparing the technical library with work-order history. A search result by itself cannot establish that the relevant documentation is absent; the analysis needs some account of what exists on both sides. The agent works across document information, diagnostic codes, and warehouse queries to investigate the gaps.

For this exercise, Blumenfeld explicitly disables the Neo4j CLI so the predefined shapes remain visible. With unrestricted CLI access, the agent might prefer custom Cypher instead. The intended investigation compares field codes, repair choices, and comeback history against the library: repeated unsuccessful repairs can suggest that the correct procedure is not being applied, while documentation with no corresponding field activity raises a different coverage question.

The live agent takes a different route from the one requested. It performs extensive semantic-expansion full-text searches instead of using the outline. Blumenfeld reports procedure mismatches and two field DTC codes with no code-level documentation. The hierarchical URIs still help it navigate the search results, but the demonstration does not show the intended outline traversal. Blumenfeld attributes the extra delay to the comprehensive searches and suggests that following explicit links could have reduced the work. A tool being available does not mean the agent will choose it.

The final demonstration asks for common patterns across bulletins and recalls, together with how many cars each affects. The agent calls Themes to identify groups, retrieves fixes from warehouse history, and associates the work orders with the relevant themes. This connects the library’s subject structure to operational activity.

A question about when the shapes are defined exposes the design sequence: first specify what the agent should receive, then derive the queries needed to produce it, and use those queries to inform the data model. The underlying document graph remains the HAS containment tree plus cross-links. The completed answer uses Themes and Connections without search, grouping cars and work orders under theme types and presenting percentages.

Asked how accuracy holds up as the knowledge base grows, Blumenfeld says he has not benchmarked this exact implementation. Customers evaluate their own ontologies and query patterns against alternatives such as ordinary vector search on their own datasets. These exercises establish the mechanisms and show the tool choices made during the live run; evaluating accuracy and improving the skill’s routing guidance remain application-specific work.

The last architectural question uses financial statements and CEO commentary as another example of structured and unstructured sources. In this workshop, the warehouse metadata graph and the document graph have no explicit edges connecting them. The agent extracts a part or diagnostic code from one source and uses it to look up information in the other, performing the cross-source match at query time. Stored links could make that mapping more deterministic. Blumenfeld leaves them out to keep the initial implementation easy to build and model-agnostic.

The online course remains available for repeating the exercises. The shared credentials are temporary: Blumenfeld plans to withdraw the Anthropic key and leave the BigQuery key available only for a while. Anyone returning to the workshop should be prepared to supply their own access.