Deploying AI Inside a Bank Is Not an Engineering Problem — Paden Gayle, Applied AI Engineers

A document exception assistant reviews client documents and drafts exceptions for an analyst. The demo makes its usefulness obvious. Then security asks where it will run. The data owners ask who authorized this use of the documents. Risk asks who will catch a wrong exception before it reaches the next system. Each answer could change the design—and the team has already invested in a design those decision-makers have never seen.

This is Paden Gayle’s hypothetical bank deployment, rather than a claim about a shipped project. Drawing on work across Bloomberg’s trade infrastructure and data platform teams and Google’s security and deployment work, he uses it to explain a familiar gap: “the demo worked, the model was fine,” yet there was no credible route into production. The practical question is what to discover in week one so the system being built is also a system the institution can operate.

Start with four kinds of deployment decisions. For each, find a person who can explain what must be decided, what evidence they need and what happens next:

• Data: Identify the source owner, the person who can grant access and the person who can approve the proposed use. These may be different people. Being able to assemble a dataset does not establish permission to use it in an application.
• Model risk: Find out which review applies to the intended use, ask to see a previous submission and identify the person who will own the system after launch. That owner will need evidence they can continue maintaining.
• Third-party risk: Draw the actual dependency chain, including the model provider, hosting arrangements and your role in operating the service. Separate relationships already reviewed from changes that need another decision.
• Audit: Ask what would let someone reconstruct a consequential action: the input, configuration, output shown to the reviewer and approval of the action. Those questions determine what the application must record from the beginning.

Finding these answers late can force changes to the deployment environment, a data path or the actions the application takes. Gayle’s Bloomberg data-platform work made the data question concrete: before designing retrieval, identify the person accountable for each source and confirm both its allowed use and where processing may occur. A change in permission is a change in architecture. This is the substance behind his phrase “discovery is the engineering.”

What does the architecture depend on besides its technical components? Consider the two rows in Gayle’s example: documents go into a model, and an analyst reviews the output; underneath are the data owner, platform owner and people who can confirm the workflow’s risk requirements. A question mark in that second row means the design depends on a decision nobody has yet located. Walk through the map with a sponsor, then confirm the answers directly with the named people. The real map may be many-to-many: one component can need several approvals, and one decision can affect the whole system. The useful result is that an unnamed owner becomes a task someone is responsible for resolving.

A discovery call should therefore produce names, decisions or next conversations. Ask where the data lives and who can approve reading it; who stopped a similar project and why; which existing controls might apply and who owns them; where production will run and who administers it; who signs the risk review and what they have signed before. Ask, too, which obligation worries the sponsor personally and who can explain it. “Security will be fine” needs a follow-up: who will review the design, and when can the team meet them? Send each owner a short description of the proposed use and confirm what they need. By Friday, the people whose decisions matter should know the project exists.

When a customer cannot provide a complete specification, the existing work supplies a starting point. Gayle recalls a Bloomberg cash-flow system whose work had to reconcile: following the flow and the people responsible for it exposed what the system had to do. For an onboarding assistant, sit with an analyst. Watch which systems they open, where they make judgments and when they ask permission. Ask what happens when a document is missing. Write down the actual sequence; that becomes the beginning of the specification.

The exception assistant begins as three lines: review the documents, draft the exceptions, analyst submits. Each line needs decisions attached. Which documents can the analyst access, and does that permission cover this processing? Should the model return a highlighted passage, a draft form or both? Who checks the draft, and who submits it? A proposed first version highlights a likely exception and prepares a form. The analyst checks the source, edits the form and submits. That is concrete enough for someone to correct.

The proposed success criterion is less review time with no increase in missed exceptions. It still needs an agreed measurement against the current workflow; it is not an achieved result. Take both the workflow and the criterion to the sponsor and control owners. They may remove a document type, change the reviewer or reject the proposed output. Each correction narrows an open question into a decision the implementation can follow.

Growing the assistant reveals why scope cannot be described only in terms of model features. Each increment introduces a different question:

• Summarize an existing document: The analyst already has access, but that access still needs to cover the proposed processing.
• Read a second system: Another data path appears, requiring someone to approve it.
• Draft a customer-facing response: Someone must check the content and establish which review applies.
• Send the response automatically: The team needs an owner for the action, a plan for an incorrect send and a way to stop and recover the workflow.

These are different engineering scopes. The applicable reviews depend on the institution and intended use, so the owners must confirm them. The useful scoping question is which increment creates enough value today. Drafting inside the analyst’s existing workflow might solve enough of the problem for a first release—provided the owners confirm that the existing controls fit.

Autonomy follows the same discipline. Start with the least authority that solves the problem, identify the consequential action and name its owner. In the exception workflow, submission is that action. Review must give the analyst source evidence and visible uncertainty, because “a review button only helps if the person can actually check the work.” Gayle connects this to cash-flow automation: automated steps need controls, and someone remains responsible for the numbers.

The autonomy ladder distinguishes who decides and who acts:

• Suggest: The analyst receives a recommendation and chooses what to do.
• Draft: The system prepares the work; the analyst checks and submits it.
• Act with approval: A person authorizes each action before it occurs.
• Act: The system operates within previously approved limits.

Moving upward changes the consequences of an error and the controls that must work. Begin at the lowest useful level, then observe overrides, recurring failures and whether people can recover when something breaks. Those observations inform the next decision about authority.

Return to the initial request: an agent reads client documents, decides which fall outside policy and files exceptions automatically. Its judgment can become an operational action without an analyst checking each case. Now keep the model and change what happens after its judgment. The assistant highlights a likely exception, points to the relevant source and prepares a draft form. The analyst checks the evidence, corrects the form and decides whether to submit.

Where does human review intervene? The model–human–submit sequence puts a check between the proposed exception and the consequential action. That revised design goes back to the discovery owners: can it use the current analyst workflow, does approved access cover the processing, which controls apply and what evidence do reviewers need? Human review neither guarantees approval nor determines a regulatory category by itself. Its practical value here is a narrower operating scope in which the team can collect evidence about errors, overrides and recovery before asking for more authority.

Evaluation becomes part of the operating plan. Gayle brings the questions from his Google security and deployment work into the AI workflow: who notices a failure, what stops, who takes responsibility and how long recovery takes. Intended use, a threat model, failure modes and a response plan should lead to a concrete demonstration of controls working when the model gets something wrong. Name the person who receives the alert and can stop the workflow.

Different failures need different detection and fallback mechanisms:

• Invented exception: The analyst checks the cited source passage and rejects the draft.
• Unreadable extraction: Stop processing and request a clean file.
• Missed exception: Use targeted offline cases to expose the weakness and keep affected work under manual review. Offline tests do not detect every live miss.

For each failure, record the detection path, fallback and responsible owner. An analyst can check an allegation against a source passage; a missing allegation requires a different strategy. Treating both as one accuracy number would obscure the operational difference.

What should a reviewer be able to inspect? The evidence pack organizes the answer into five files:

• Intended use: Users, operating boundaries and prohibited actions.
• Data lineage: Each example’s source and approval for its use.
• Failure modes: How failures are detected and what fallback follows.
• Results by segment: Coverage and errors within relevant groups.
• Controls: Reviews, thresholds, stopping rules and owners.

Regenerate the evaluation evidence with each run, while preserving the original input, prompts, model version, configuration, output and reviewer action. That record lets someone reconstruct the historical decision. Calling the model again later may return a different answer; a rerun cannot stand in for what the analyst actually saw and approved.

The fourth file needs more than an aggregate result. For the hypothetical workflow, separate groups such as clean documents, scans and new document types. Measure errors and control behavior within each, then lead with the weakest result. An average can conceal a serious gap, while a group with little evidence may support very little confidence at all. State that limitation and restrict that use until enough evidence exists. The restriction is part of what the results support, rather than an inconvenient footnote to an overall score.

The first week closes with five questions:

1. Is the data owner engaged and aware of the project?
2. Are the production environment and its administrators named?
3. Can the sponsor describe the workflow without saying “AI”?
4. Can the smallest useful scope fit existing controls, confirmed with their owners?
5. Do you know the risk signer and the experience of the previous sign-off?

Use the answers to plan the next week. Four or five yeses suggest a credible path to ship. Three means the unresolved decisions need owners and dates. Zero to two means resolve those decisions before committing to delivery, potentially by shrinking or redefining the scope. This score does not predict approval; it identifies where the remaining work sits.

The closing instruction is practical: draft the specification, shrink it with the people responsible, give the system only the authority it needs and build the evidence pack in sprint one. A working model leaves all of those tasks ahead of the team. “We go find the people” is how the hypothetical assistant begins acquiring a route into production.