What Breaks When You Build AI Under Sovereignty Constraints

How much control does your organization actually have over its AI system? Bilge Yücel opens with sovereignty as the organizing question. Introducing herself as a senior developer relations engineer at deepset, she places it in the context of the company behind Haystack and its enterprise platform. She names Airbus, Bosch, Siemens, the European Commission, and German ministries as examples of the organizations deepset serves—settings where control over an application extends beyond choosing its model.

Sovereign AI means being able to design, deploy, and operate AI systems on an organization’s own terms. For an engineer, that becomes explicit control over data flow, model choice, infrastructure, observability, and operations. The useful question is not simply whether a system is sovereign, but which decisions the organization can make and enforce itself.

These boundaries overlap, but each exposes a different dependency. Keeping storage local, for example, says nothing by itself about where inference runs or who can change the application.

Start with a European organization whose data must remain within a trusted processing boundary. Its database may be in Europe, but sending documents to an embedding API hosted in Virginia moves their processing outside that boundary. Storage location does not determine the whole data path. Yücel frames this through a broad GDPR residency statement; the legal distinction is that EU rules permit international transfers under mechanisms such as adequacy decisions and appropriate safeguards. The endpoint example establishes a boundary crossing, not enough information to establish a GDPR violation.

Geography is only one part of data control. A colleague who can retrieve information they are not authorized to see also breaches the intended boundary, even if every database and model runs in the approved jurisdiction. Access permissions therefore belong in the same analysis as storage and processing locations.

Next, locate the application itself: ingestion, RAG pipelines, agents, and their tools all execute somewhere. Yücel describes infrastructure choices as a spectrum from maximum control to maximum convenience: air-gapped environments, private VPCs, sovereign cloud, and SaaS. Sovereign-cloud properties depend on the provider. The slide’s air-gap/EU AI Act and VPC/GDPR safety labels are simplifications, not compliance guarantees; deployment isolation does not replace the documentation, oversight, and other obligations described in the AI Act guidance.

Provider jurisdiction adds another dimension. European hosting and a provider’s exposure to US legal process are separate questions. In Yücel’s SaaS example, a US-headquartered provider creates a concern even when the application and data remain in Europe. More precisely, the CLOUD Act concerns potential compelled disclosure by providers subject to US jurisdiction of data in their possession or control, regardless of storage location. It does not create unrestricted access merely because a company has US headquarters.

Model sovereignty begins with the freedom to choose and switch models. A system that works with only one provider inherits that provider’s outages and price changes. Supporting alternatives in principle is not enough: if switching requires extensive changes to application logic, the organization cannot exercise its choice readily. Swappability is an architectural property, not just a procurement option.

Training-data provenance is a separate concern. Yücel acknowledges that teams often cannot determine which data trained a model. She suggests European providers have an advantage in this context, but a provider’s headquarters does not establish the origin of its training corpus. That uncertainty remains even when inference is under local control.

Operational sovereignty extends control into production. It includes monitoring and evaluating behavior, inspecting model inputs and outputs, and managing both model and application updates in a controlled, auditable way. In Yücel’s HR and finance examples, human oversight is part of that operating model. Building the system is only the beginning; the organization must also be able to maintain and govern it.

The required degree of control varies. A high-stakes finance or healthcare deployment may need an air-gapped environment, while another enterprise or startup may not need the same restrictions across every pillar. The practical task is to identify the control the system actually provides and the vendor dependencies it retains.

Now consider the Monday-morning request: the organization already has a working AI system, and a manager or CIO asks the team to make it sovereign. The intuitive first move is to replace its frontier API with a self-hosted model. That immediately exposes more than an endpoint dependency. API logic must be adapted, prompts may need revision, and the system’s performance must be evaluated again. Moving inference changes assumptions embedded throughout the application.

Moving private data into the required jurisdiction creates another set of dependencies. What looked like a storage relocation can leave the team operating multiple databases and instances. Retrieval must now decide where to search:

• Classify and route: determine which database should receive a query before searching.
• Search multiple databases: send the request to both and gather information from their results.

Either approach introduces application behavior that a single-store design did not need.

Replacing managed infrastructure with an on-premises deployment exposes work the cloud provider previously absorbed. Kubernetes cluster management becomes the team’s responsibility. Hardware constraints become explicit too: the application may run on CPUs while model inference runs on GPUs, requiring network and connection management between them. Changing the deployment boundary changes who owns these operational problems.

Finally, adding observability reveals whether the application was understandable in the first place. An opaque AI layer cannot become auditable merely by attaching a logging destination. The team must identify what happens inside it, decide what to record, and establish version control for the application and the system it operates.

Haystack addresses part of this migration through consistent component interfaces. A cloud-backed component can be replaced with one connected to self-hosted inference while preserving the surrounding interface. Yücel describes integration changes as potentially small, leaving the team to concentrate on hardware and connectivity. That is a claim about reducing integration edits, not eliminating prompt evaluation or completing an entire migration automatically. Orchestration cannot supply missing GPU capacity.

The next benefit is explicit data flow. Typed, declared inputs and outputs make it possible to inspect a pipeline definition and follow what moves between components. Agent execution is less deterministic, but its tool calls and tool outputs can still be traced. A changing execution path need not be an invisible one.

Haystack also supports serializing pipeline definitions to YAML. Committing those definitions makes application structure inspectable across revisions: a previous commit shows the configuration that was recorded then. This versions the pipeline definition; it does not, by itself, capture every model weight, external dataset, or dependency needed to reproduce execution. Open-source components add another form of control: teams can inspect, customize, and extend the implementation rather than treating it as a black box.

The proposed agent architecture begins with an input guardrail. It checks for prompt injection and, where relevant, whether the request fits the application’s permitted regulatory or task-specific intent. Input classified as unsafe leaves the application path; input classified as safe proceeds to the agent. This is a classification control, not proof that every malicious request will be detected.

The agent combines an LLM, a system prompt, and tools. Those tools can call APIs, search a knowledge base, invoke other agents, or connect through MCP servers. After the agent finishes its work, an output guardrail checks the proposed response for compliance and sensitive-information leakage before it becomes the user-facing output. The design goal is to keep each part replaceable and its behavior traceable at the required level of sovereignty.

The same architecture can mix deployment choices by task. Public-data work may continue to use proprietary hosted models. Guardrails, knowledge-base tasks, or agent inference may instead use locally hosted models. Yücel names Mistral, Google, NVIDIA, and Jina as possible sources of models for these roles; she does not require one provider or one deployment policy for every component.

Component activity and agent execution can feed spans into an LLM observability system, while OpenTelemetry integration permits custom observability. Available instrumentation must still be configured deliberately: current Haystack tracing documentation requires explicit activation and disables input/output content tracing by default to avoid exporting sensitive information. Storage is another independent choice; providers offering both cloud and open-source editions can support an on-premises deployment when that boundary is required.

The implementation walkthrough starts with a guardrail model configured through NvidiaChatGenerator. Connecting it to LLMMessagesRouter turns classification into pipeline routing: the demonstration describes safe and unsafe paths. The router matches model outputs against configured patterns and also has an unmatched output, so an implementation must account for classifications that fit neither expected route.

Next, MCPToolset connects to a locally hosted MCP server and selects named tools. The examples are knowledge-base search and PDF-report generation. Selecting tools matters because the server may expose many capabilities that this particular agent should not receive. The tool_names option restricts the discovered inventory; it is not a substitute for authorization enforced by the server.

The inventory can also contain tools built from Python functions, Haystack components, other agents, or complete workflows such as ingestion and RAG pipelines. After assembling these capabilities, the walkthrough places them in a SearchableToolset. It uses BM25 to discover relevant tools dynamically, avoiding the need to put every definition into the model’s context. Yücel’s inventory of hundreds of tools is a hypothetical motivation, not a measured scale or performance result.

The agent’s system prompt states its purpose and the tools it can use. Calling it a sovereign agent describes the intended role; enforcement comes from the surrounding controls. Its chat generator supplies inference. For an internal on-premises endpoint, a team can create a custom component that calls the company’s inference service. If the endpoint is OpenAI API compatible, an existing Haystack component can provide the connection.

After connecting inference and tools, the walkthrough adds confirmation strategies. Its payment example distinguishes two kinds of action:

This distinction keeps a consequential submission behind repeated approval while allowing repeated retrieval after permission. The current Agent API reference is not a historical specification of the demonstrated confirmation strategy, so the example should not be read as establishing the exact scope of cached approval.

The final assembly brings the controls into one pipeline:

1. Add a tracer connected to the LLM observability system.
2. Add the input guardrail.
3. Connect the agent and its tools.
4. Connect the output guardrail.

The concrete request is to retrieve outstanding payment requests for Q3 and generate a PDF. The walkthrough describes the agent using its tools to create the report and save it to a directory. This example brings routing, tool access, inference, and tracing into one application flow; it does not establish a deployment procedure or a measured performance result.

The closing test returns to capabilities the team can demonstrate:

• Model replacement: Can you swap models without changing application logic?
• Auditable execution: Do you have reproducible run logs stored in a compliant way?
• Incident response: Can your team respond to an incident without calling a vendor or hyperscaler?

These questions put the burden on the working system: whether the organization can change it, reconstruct its behavior, and operate it when something goes wrong.