Skill Issue: How We Used AI to Make Agents Actually Good at Supabase

How do you give an agent the product knowledge it needs, and then check that the knowledge changes its behavior? Pedro Rodrigues’s workshop approaches that question through the work of writing Supabase skills. Onstage, he renames the session Level Up Your Skills, reserving Skill Issue and the production lessons for the following day.

The presentation itself is a small demonstration: a localhost application built with Claude, complete with a dark-mode toggle. Rodrigues introduces himself as a Supabase AI tooling engineer working on agent experience, or AX—the counterpart to developer experience. The workshop follows a practical sequence: write a skill, test it manually, and automate the test with evaluations.

A skill is a folder, not merely a long prompt. Its required entry point, SKILL.md, contains Markdown instructions and frontmatter with a name and description. Optional files can supply reference material or executable workflows in Bash, Python, or another language.

The central mechanism is progressive disclosure. Initially, the agent receives the skill’s metadata rather than its entire body. The description acts like an envelope: it tells the agent enough to decide whether to open the instructions. Once loaded, those instructions can point to additional files that the agent reads only as needed. The Agent Skills specification describes this separation between discovery metadata, instructions, and supporting resources.

Rodrigues compares SKILL.md to a book’s index with extra instructions. Reference files are ordinary Markdown chapters, often kept in a reference directory. They can link to one another, so a skill can form a graph of information; that possibility does not require building a deeply nested reference hierarchy. The useful design property is that a task can reach the relevant detail without loading every chapter first.

Skills and MCP address different parts of an agent workflow. A skill supplies context and instructions, including workflows too extensive for a tool description. MCP supplies an integration through which the agent can act. An agent without Bash access can still use a service through an MCP tool.

A remote MCP tool does not require the agent’s machine to provide the tool’s execution environment. A bundled script does: its commands and dependencies must work on the local operating system. That distinction matters more than treating skills and MCP as competing packages.

Testing Markdown means testing what the agent does after reading it. The scope can be narrow, like checking a tool call, or broad, like checking a complete workflow. Because an LLM produces variable outputs, exact text matching often captures less useful behavior than inspecting the steps taken, tools called, and resulting state. An evaluation still starts with an input and an expected outcome.

The cycle Rodrigues adopts comes from OpenAI’s Testing Agent Skills Systematically with Evals:

1. Define success. Decide whether the skill should route the agent to documentation, enforce a workflow, or produce a particular result.
2. Write the skill. Create SKILL.md, adding references and scripts where they help.
3. Run scenarios. Exercise the skill manually or through an evaluation runner.
4. Grade behavior. Compare outputs and actions with the intended criteria.
5. Iterate. Change the skill and repeat the checks.

Braintrust is introduced as a way to observe agent behavior inside controlled evaluation scenarios. Even when the overall workflow is nondeterministic, individual checks can remain deterministic: a tool either was called or was not, and a database object either has a required property or does not.

The testbed is a deliberately vulnerable performance-review application with four switchable accounts: an employee, two managers, and an HR representative. The plan is to expose a database mistake manually, add a skill that guides the agent away from it, and reuse the scenario in an automated evaluation. Supabase provides the backend: Postgres, authentication, storage, and edge functions, which Rodrigues compares to AWS Lambda functions. He introduces the service as an open-source alternative to Firebase.

The workshop repository contains the employee-directory exercise. In the recording, Rodrigues works in a workspace with two Next.js apps, including the presentation; the application he opens is under demo. The current public checkout has a different root layout, so use its README for setup rather than assuming those recorded paths still apply. He credits skills with improving the presentation’s visual quality before turning to the application itself.

The application lets HR inspect employee information and lets the presenter switch users to test access boundaries. The planned Reports page initially includes salary and average performance ratings by department. The actual task sent to Claude is narrower: create department_stats, a SQL view showing headcount and average salary by department. A prepared reporting interface will display the database result.

The local setup follows a short sequence: clone the repository, install dependencies, start Supabase through npx, reset the database to its migrations and seed data, then run npm run dev. The dashboard is served at localhost:3000/dashboard. A prepared .mcp.json points Claude Code at the local Supabase MCP server, which requires no authentication in this setup.

Rodrigues reports 20 tools in the local MCP server and tentatively recalls about 29 in the production server. The relevant capabilities are table listing, SQL execution, apply_migration, and database advisors. Claude first inspects the schema, then uses apply_migration to create department_stats, aggregating profiles by department. It encounters an error, retries, and tests the query.

The database result exists, but the Reports interface does not initially show it. Rodrigues sends another prompt to connect the view to the Next.js application, keeping direct SQL queries as a fallback for testing different users. Claude finishes the interface work and reports that the feature is working. That establishes that the query returns data and the interface can display it; the next check is who can see that data.

Logged in as Julia from HR, the report looks plausible: Engineering has five people with an average salary of 107K, HR has one person, and Product has four people. HR is supposed to have a company-wide overview. Managers should see only their departments, and employees should not gain access to other employees’ records. Switching to Bob, head of Engineering, exposes the problem: he can also see HR and Product information.

Functional correctness has concealed an authorization failure. The database migrations already enable row-level security, or RLS, on profiles and performance_reviews. RLS filters access at the database boundary rather than relying solely on application code. Rodrigues inspects a policy involving reviewer_id and current_setting, then turns to the permission context of the newly created view.

By default, underlying access through a Postgres view uses the view owner’s permissions and applicable RLS policies. In this setup, that allows the view to bypass the restrictions intended for the caller. The security_invoker option changes those checks to use the invoking user. It does not copy policies onto the view, and a default view does not universally bypass RLS; the owner and permission context matter. The PostgreSQL 15 view documentation explains this distinction.

The prepared Supabase Security skill contains a short security checklist. Its view example is deliberately tailored to department_stats: create the view with security_invoker enabled. The option is available in Postgres 15 and later. For an existing view, the focused SQL change is:

ALTER VIEW public.department_stats
SET (security_invoker = true);

This changes whose permissions govern access through the view. The underlying table permissions and policies still have to express the intended employee, manager, and HR boundaries.

The description begins with the verb “use.” Rodrigues reports that action-oriented wording, especially that verb followed by the skill’s purpose, has made Claude more likely to load his skills in his experiments. He does not establish why it works or whether the observation transfers to other models. The body also directs the agent to enable RLS on tables in public or otherwise exposed schemas. Direct frontend access needs those database policies; without them, the application must supply the filtering itself.

Rodrigues installs the skill with the Vercel Labs skills CLI. It accepts remote repositories and local paths beginning with ./, and supports multiple agent harnesses. An audience question about project boundaries leads to an explicit choice of project installation rather than global installation. This differs from the harness-specific plugin packaging he describes for Cursor and Claude Code.

The installer detects supabase-security, asks which agents should receive it, and Rodrigues selects Claude Code. Project installation creates the skill under .agents and symlinks it into .claude/skills. A fresh Claude Code session can then discover the installed skill.

There are three activation paths: let the description trigger automatic discovery, explicitly ask to use the named skill, or invoke it through Claude Code’s slash command. Rodrigues describes the named request as highly reliable in his experience and chooses explicit invocation for the demonstration. Before rerunning the task, he resets the disposable local database:

npx supabase db reset --yes

The reset reapplies migrations and seed data. It also removes the earlier view because the agent’s MCP migration changed the running database without creating a corresponding local migration file.

An audience member raises the obvious tension: if skills promise automatic discovery, why must the user name one? The agent still decides when to load it. Rodrigues recommends building both positive scenarios, where the skill should load, and negative scenarios, where it should remain unused. Run those prompts through the CLI, inspect whether activation occurred, and iterate on the description. That tests discovery behavior, independently of whether the instructions help once loaded. He treats the surrounding standards and context-management approaches as still evolving.

After the reset, the first attempt does not create the view. Rodrigues revises the request, the skill loads, and Claude resumes the familiar workflow: inspect the tables and prepare a migration. This time the migration includes security_invoker. The skill has changed a concrete implementation decision.

The application check still fails. With Alice Chen selected as an employee, department reports remain visible. Rodrigues suspects another policy problem and asks the agent to troubleshoot. The visible change in SQL therefore does not establish a completed authorization repair.

Supabase’s database advisors offer another check for exposed data and potential vulnerabilities before production. Rodrigues proposes adding advisor execution to the skill if the agent cannot diagnose the issue itself. He treats SKILL.md as an on-demand prompt template: it can require an additional diagnostic step as well as a SQL option. The agent offers further migration work, including a more complex change involving a separate table, but the walkthrough moves on without a verified fix.

An audience request to inspect /context reveals a crowded environment with multiple skills and MCP servers. The displayed skills category occupies about 1.3K tokens across several installed skills, not just Supabase Security. Clearing the session does not produce a useful isolated before-and-after comparison for this small skill.

The inspection also qualifies the earlier contrast with MCP: the newer Claude Code version in the demonstration has tool search, which loads MCP tool definitions on demand. Skill packaging provides progressive disclosure through its own structure; tool loading also depends on the client. Rodrigues describes broader protocol support as something he expects from a forthcoming MCP roadmap presentation, rather than a capability established by this workshop.

A second question makes the distinction practical: how should an agent obtain enough of a very large database schema to write a query? Rodrigues separates the information the agent needs from the mechanism that retrieves it. A script or MCP tool can extract schema information. For a remote production service, he favors an MCP integration that handles execution and authentication outside the agent’s local shell, keeping authentication tokens out of the agent’s working instructions.

The retrieval tool must actually support partial results. A skill cannot make an all-or-nothing schema endpoint incremental merely by asking politely. Rodrigues suggests a chunking parameter—using buffer as an example—and a skill that explains which tool to call and how to request successive portions. The tool provides the retrieval capability; the skill supplies the procedure for using it.

The manual comparison has a limited but useful result: the first migration omitted the flag, and the skill-guided migration included it. Rodrigues attributes the unresolved application behavior to the additional vulnerabilities seeded into the exercise. To maintain the skill, however, he needs more than a successful local interaction. Changing one checklist item should not silently break another workflow, and the checks need to travel across agent environments.

The hardest evaluation work is defining representative scenarios and their expected behavior. Prompts and expectations can live in CSV or JSON; Braintrust and Langfuse can add observability. For this workshop, Rodrigues uses an eval.json containing an array of scenarios, following the Agent Skills guidance he cites. The format is the workshop’s test organization, rather than a requirement imposed by the skill file format.

Two grading approaches serve different needs:

• Deterministic assertions: Check observable facts, such as whether a tool was called or a view has the required setting.
• LLM-as-a-judge: Give another model the output and explicit success criteria, then ask it to grade the result.

The workshop reuses the department-report prompt and expects security_invoker=true on department_stats. Its prepared runner uses deterministic assertions for that check.

A Python script orchestrates the comparison:

1. Reset the local database so each condition starts from the same migrations and seed data.
2. Run Claude Code headlessly with the evaluation prompt, using the participant’s own Anthropic credentials.
3. Run a with-skill condition that appends SKILL.md to the system prompt.
4. Run a without-skill condition and retain the outputs for comparison.

Appending the text tests what happens when the instructions are present; it bypasses automatic skill discovery. Rodrigues proposes a more realistic alternative: put the skill in .claude/skills inside a fresh Docker environment and let Claude Code discover it naturally.

The live runner initially fails to find the skill because of a naming mismatch. After Rodrigues corrects the name, it runs the with-skill condition first, then the without-skill condition, writing their outputs into an iteration workspace. The next task is to inspect both the generated result and the grading record.

The with-skill output contains security_invoker, but grading.json marks the check as failed. Rodrigues traces the discrepancy to his assertion logic involving pg_class and reloptions. The walkthrough does not resolve the exact defect. The immediate debugging lesson is to compare the assertion’s evidence with the generated artifact before concluding that the agent ignored the skill.

The without-skill run then supplies another surprise: it also includes the flag. Rodrigues reports that the grader fails the with-skill condition and passes the without-skill condition even though the inspected outputs are effectively the same. The displayed with-skill grading file records zero passes and four failures. Those grades are disputed results from the live harness, not evidence that the skill reduces correctness. A single variable model run, combined with a faulty check, cannot establish a reliable improvement rate.

There are two distinct failure modes here. Deterministic grading code can inspect the wrong object or encode the wrong expectation. An LLM judge can additionally produce an unreliable judgment. The live discrepancy belongs to the first category; Rodrigues raises judge hallucination as a separate risk when discussing how evaluation becomes harder with model-generated behavior.

The first evaluation pipeline can remain simple: compare with-skill and without-skill conditions on the harness that will actually use the skill. A Docker container or sandbox gives each run a fresh environment containing only the skill under test. That removes unrelated installed skills and leftover state from the comparison. The workshop stops at this testing structure; Supabase’s product-skill maintenance and production evaluation are reserved for the next day’s talk.

The final audience question asks whether progressive disclosure means one can keep installing skills indefinitely. Rodrigues offers no numerical limit. For local experimentation, he tolerates a messy inventory: unused descriptions still consume context, but their small size makes aggressive pruning less urgent in his personal workflow. For production and CI, his advice is stricter: include only the skills needed for the specific task.

A production skill is maintained documentation with behavioral consequences. Publish it alongside the repository, update it when features or workflows change, and put that maintenance requirement in CLAUDE.md or AGENTS.md. Periodic jobs can check whether its workflow remains valid. Where skill-loading telemetry is available, long periods of nonuse can prompt a review of whether the skill still belongs in the inventory. The work continues after the Markdown is written: the instructions must remain relevant to the product and useful to the agents that read them.