AI Platform Engineering

Shared services and workload needs

An AI platform combines reusable services with supported workflows for identified builders and operators. Its scope follows recurring needs: connecting approved models, accessing company data, deploying workloads and investigating failures. Existing infrastructure can supply these capabilities; the platform team need not operate every underlying component. The CNCF platform definition emphasizes integrated capabilities organized around users. [1]

A model is a fitted computational component that produces predictions or generated outputs. Inference applies that fitted component to inputs; training, fitted state and inference explains the distinction. Online inference serves immediate requests, whereas batch inference processes accumulated inputs asynchronously. These execution modes can share model artifacts without sharing a response-time contract. [2]

Two teaching workloads expose the difference. A document-grounded assistant retrieves permitted evidence and answers a person waiting for a response. A classification job assigns categories to accumulated documents. Both need model access and diagnostics, but only the assistant uses a retrieval index. The following requirements are design choices, not reported service measurements. [3][2]

Shared model access does not transfer responsibility for business logic, consequential actions or correct outcomes. Application teams retain those duties. Similar requests can also conceal different policies, as Customer variation and reusable capability explains. The workload table makes these differences explicit without implying that both applications should adopt an identical implementation. [6][8]

Configuration, execution and service contracts

The control plane manages configuration, policy and resource intent. The data plane performs the primary work: processing requests and executing jobs. These terms classify operations, not entire products. Updating gateway routes is control-plane work; forwarding a model request is data-plane work. Control-plane and data-plane definitions make this separation explicit. [9]

Desired state records what should exist; observed state records what components report. Reconciliation compares them and requests changes. A controller can request execution without executing the workload itself. Consequently, accepting configuration revision R2 while an executor reports R1 establishes pending convergence, not active R2 behavior. [10]

An agent harness is the application runtime that dispatches operations, preserves execution records, enforces limits and checks results. Shared scheduling or storage can support many harnesses, while application-specific state and completion rules remain local. The harness and its execution contract covers that runtime boundary; platform control should not depend on a model deciding to obey it. [11]

Tenant identity and delegated access

A tenant is an administrative and isolation unit, such as a team, product or customer. It is distinct from a person and from a worker process. A workload identity identifies executing software. Multi-tenancy guidance separates restrictions on resource access from restrictions on resource consumption; a tenant name alone establishes neither. [13]

Authentication establishes identity; authorization permits an action on a particular resource. Least privilege grants only the access required. A confused deputy is a privileged service induced to use its authority for an unauthorized caller. A shared worker with broad storage credentials must therefore check the caller's narrower authority before reading documents or publishing results. [12][14]

Bind tenant and environment from verified identity, not a model-generated field or unvalidated header. Preserve that binding in queued work and downstream requests. Resource lookup must combine resource identity with authorized tenant context, including for artifacts, caches and diagnostic records. Authorization at access and disclosure boundaries explains why a permitted read does not automatically authorize every later disclosure. [15][12]

Delegation carries constrained authority to another executing component. Uber describes exchanging credentials at each hop for short-lived, recipient-specific tokens while preserving the human and agent actor chain. Recipients verify the token's intended audience, and a gateway applies tool-access policy. This supplies a concrete mechanism for shared workers to act with attributable, bounded authority. [16]

Sharing a connector also requires an explicit audience decision. In Viktor's reported incident, an administrator connected personal email as a team integration, making messages accessible through the shared agent. The product subsequently added integration scoping. Reusing credentials reduced setup work, but the integration's permissions exceeded the intended sharing boundary. [17]

Credentials can remain outside generated code while a trusted broker performs permitted operations. Cloudflare's documented egress mechanism illustrates this separation: direct outbound calls can be blocked or mediated, and the broker can attach provider credentials. Granted bindings remain separate access paths. The broker's actual resource and operation checks determine the authority exposed. [18]

Model gateways and capability contracts

A model gateway mediates inference requests between callers and model backends. It can centralize identity, provider authentication, routing and usage records. Aperture's gateway description illustrates these operations while distinguishing supported API formats. One entry point simplifies integration without making its backends behaviorally interchangeable. [19]

Compatibility checks must precede fallback. OpenRouter documents parameter-support filtering and separate fallback and data-policy controls; default routing can otherwise ignore unsupported parameters. These are partial mechanisms, not a complete application contract. Output parsing and semantic validation remain necessary, as Completion, parsing and runtime validation explains. [20]

Provider-executed tools make the distinction concrete: the provider runs the operation and returns its result, reducing application infrastructure while creating a provider dependency. A custom tool instead leaves execution with the application. A common SDK function cannot erase that difference in responsibility. [21]

Record the requested alias, resolved provider/model and routing revision for each call. An alias is a selection name; its target can change. A serving backend must be a complete executable model instance, even when spread across devices. Serving replicas, routing and request state explains that execution boundary. [22][25]

Workload admission and observable completion

Acceptance records submitted work. Admission permits execution under current limits; placement assigns physical resources. Running and completion require later observations. Kueue, for example, distinguishes quota reservation from scheduling. Its physical-capacity checks depend on configured mechanisms. Neither an accepted API call nor reserved quota alone proves that a worker is running. [26]

A long-running operation gives clients an identity they can inspect after submission. Define progress as observable work, such as documents completed, rather than an unsupported completion percentage. Declare partial-result visibility, durable result location and operation-record expiry. Long-running API guidance separates immediate rejection from failures recorded after execution starts. [4]

Cancellation requested and execution stopped are different observations. Cloud Speech's cancellation API is explicitly best effort: successful acknowledgement requires a later status inspection. The operation can complete instead. Durable stopping also preserves unresolved work and useful partial outputs; Stopping and observed termination covers those runtime responsibilities. [24][5]

Artifact identity and release dependencies

An artifact is a retained object used or produced by work. An artifact registry catalogs versions, metadata and storage references. MLflow's registry distinguishes numbered versions from mutable aliases. Reassigning an alias affects later resolution; it does not prove replacement of already loaded instances. [25]

A content digest identifies bytes through a cryptographic hash. Comparing it with a trusted reference detects changed content; a filename or URL cannot do that. Authenticated update metadata can additionally identify authorized publishers and reject stale updates. Neither a matching digest nor a valid signature establishes that the artifact is safe or useful. [29][30]

Provenance records origin and production history; lineage connects derived artifacts to inputs and producing activities. The W3C provenance model separates entities, activities and responsible actors. Record actual dependencies, including dataset releases described in Dataset releases and reproducible lineage, rather than attaching every available platform asset to every run. [31]

The assistant's release manifest binds application/runtime code, its query encoder, prompt configuration, generation-model reference and index snapshot. The encoder converts queries into numeric vectors, or embeddings, compared with document vectors stored in the index. Equal vector lengths do not guarantee matching coordinate meanings. Dense retrieval and representation compatibility explains the required shared scoring space. Classification needs no index unless its implementation uses one. [32]

Externally hosted models may expose identifiers without exposing their weights. Anthropic distinguishes pinned model identities from convenience aliases, but also documents possible changes in surrounding routing, safety and sampling infrastructure. Recording the model identifier improves traceability without guaranteeing identical future outputs. Stored artifacts, external references and observed response metadata should therefore remain distinguishable. [33]

Data and artifact lifecycle controls

Governance owners decide permitted purposes and retention obligations; platform services implement them. Lifecycle fulfillment across derivatives separates those decisions from evidence that each destination complied. Immutable identity preserves which bytes existed, not perpetual permission to use them. [34]

Revocation removes future credential use; it does not erase stored data or undo disclosure. Correction changes information, invalidation disallows dependent output, and rebuilding creates a replacement artifact. These actions can finish at different times. A completed live-source deletion must not conceal an unresolved external copy or an active job still holding the old input. [38][31]

Retained versions complicate physical deletion. Iceberg keeps data files while retained snapshots reference them; snapshot expiration and orphan-file cleanup address different objects. Cleanup must also respect active writes: removing a file before an in-progress writer finishes can corrupt the table. A platform lifecycle controller therefore needs dependency and execution state, not just a deletion command. [35]

Backup handling needs an explicit restore contract. UK ICO guidance permits some temporary backup retention beyond live deletion when data is put beyond use pending overwrite; applicability belongs with governance owners. A platform should test that restoration reapplies current withdrawal restrictions before serving data. That test is a design requirement, not a guarantee supplied by backup retention alone. [37]

Deleting training records does not remove their learned influence from existing parameters. Machine unlearning addresses that separate problem; retraining without the affected records provides a baseline. The platform must distinguish copy-deletion completion from any required model-level remedy. [39]

Deployment specifications and readiness

A deployment specification binds a release to its operating environment. The proposed contract includes dependency identities, resource requirements, configuration, access-policy revision, credential references rather than secrets, operating limits and acceptance-evidence references. Registration supplies an identifiable candidate; deployment additionally associates it with execution resources and operating configuration. [40][41]

Startup, liveness and readiness probes answer different questions. Startup allows initialization, liveness can trigger restart, and readiness controls eligibility for traffic. A probe establishes only its configured check. It does not establish model identity, complete response delivery or semantic correctness; poorly chosen liveness checks can even amplify overload through repeated restarts. [43]

Application acceptance evidence must identify the candidate bundle and intended tasks. An evaluation score detached from its cases, dependencies and conditions cannot authorize every use. Release and revision decisions covers that methodology. Platform checks establish compatibility and operability; application checks establish whether the resulting behavior meets the intended contract. [42][45]

Deployment demonstrations can expose this distinction directly. In the Agentuity walkthrough, the deployed webhook created a session, but agent execution failed. The speaker tentatively attributed it to memory configuration. Endpoint reachability and session creation were observed; successful workload execution was not. [46]

Controlled rollout and operational intervention

A canary exposes bounded production traffic to a candidate while a control cohort continues using the baseline. Compare version-specific outcomes under declared stopping and promotion criteria. Traffic must cover relevant workloads for a sufficient observation window; a small but unrepresentative sample can miss defects. Assign promotion authority before the experiment starts. [47]

Behavior can change without an application deployment. Prompt flags, model aliases, routing rules, access policies and runtime configuration all alter execution. Attribute observations to those revisions and check capacity while old and candidate versions overlap. Shadow traffic needs separate handling: discarding candidate responses does not prevent its tools from writing to production. [48][49]

A kill switch must be checked at subsequent decision points, including in spawned agents. Resolving it only when a session starts leaves active sessions using old settings. It does not necessarily interrupt a tool already executing. Record who changed the control, when it changed and which work observed it. [50][52]

Emergency controls require drills because configuration changes can silently break dormant paths. Temporary rollout flags need owners and retirement plans; permanent emergency controls need continuing maintenance. Test relevant combinations of active settings, since individually acceptable prompt variants can interact badly. [53]

Compensation differs from restoring a snapshot. Concurrent work must survive, irreversible effects may require escalation, and the original state may be unattainable. Persist compensation progress and make repeated corrective operations safe. A routing rollback can contain further exposure while a separate application recovery process resolves completed effects. [51]

Tenant isolation and finite capacity

Authorization isolation restricts access; resource isolation limits consumption; failure isolation limits shared disruption. A noisy neighbor is a tenant whose activity degrades others. Namespace authorization and quotas help with different parts of this problem, but neither covers every shared resource. Network traffic, for example, can remain a source of interference. [13]

A quota limits entitlement; a reservation accounts for allocated resources. Ready capacity additionally requires usable execution resources. Fair admission declares which tenants can proceed when demand competes. Borrowing unused allocations improves utilization but needs return rules; preemption evicts admitted work and therefore needs explicit consequences for its state and deadline. [26]

Assume four ready execution slots, identical one-slot requests and non-preemptible batch tasks. Four batch tasks are waiting before an interactive request arrives. Unrestricted admission can occupy all four slots; reserving one for interactive work permits only three batch tasks to start. The reservation changes eligibility, not execution speed. [54][55]

Shared pools trade utilization against interference; dedicated pools make allocation clearer but can leave capacity idle and still share upstream dependencies. Headroom is capacity retained for bursts or failures. Startup delays and concurrent admissions must enter the contract: repeatedly checking an unchanged available-budget counter cannot reserve the same capacity safely for several callers. [56][57]

Usage-priced services charge for consumption; provisioned capacity incurs expense while available. Provisioning, warm capacity and admission policy covers the economics. The platform must expose the actual capacity promise and exhaustion behavior. Process, container and virtual-machine boundaries covers execution isolation; selecting a boundary does not configure suitable CPU, memory or time limits. [58][59]

Configured capacity can also drift from reality. Krea reported manually maintained queue resources becoming stale as cluster nodes disappeared or entered maintenance, disrupting gang scheduling—the admission of a required group together. Resource accounting therefore needs reconciliation with usable infrastructure, not merely a larger quota. [28]

Evidence at service boundaries

A service-level indicator, or SLI, measures a defined service property. A service-level objective, or SLO, sets its target over a window; the allowed fraction of misses is its error budget. Task metrics, latency and usage accounting supplies the broader framework. A useful contract also declares eligible work, success criteria, missing-outcome treatment and the owner of resulting decisions. [60]

Count attempts separately from logical requests. Keep rejections, cancellations and unknown outcomes visible under declared inclusion rules. Server completion can precede or outlive client delivery, so server latency is not automatically user latency. Missing evidence is unavailable coverage, not a successful event or a zero-valued measurement. [27][60]

Attribution should connect tenant, request or job, attempt, resolved model, release and policy revision. This is an instrumentation contract, not a field set supplied automatically by one tracing library. OpenTelemetry distinguishes requested and response model identities, while workflow and policy provenance need application conventions. Later feedback should retain its own criterion, source and observation time. [22][61]

Gateway records can standardize attribution: Uber describes recording caller, user, team and project context through its shared endpoint. Their coverage still follows the boundary observed. A network gateway can capture a model-visible tool request without independently establishing that the tool executed successfully or that every local action passed through the gateway. [62][63]

Successful transport can conceal incorrect evidence. One reported analytics workflow requested quarterly consumer-price data but retrieved monthly records; the final answer hid the intermediate table. The consuming application needed a frequency check. A responsive gateway or successful model call could not establish that domain condition. [64]

Diagnostic access is itself a protected service. Retain only necessary content, restrict readers and declare retention. Aperture documents a mode retaining usage metrics without writing request and response bodies to disk. That setting neither establishes upstream-provider retention nor restores individual attribution when several people share one gateway identity. [19]

Failure containment and verified restoration

A failure domain contains components exposed to a common failure. Two backends can still depend on one gateway, identity service or artifact store. Static stability means existing resources can continue without control-plane changes. It helps only when recovery avoids the failed dependency; requiring fresh provisioning or artifact downloads can make an apparently independent route unusable. [65]

Continuity needs a separate authorization rule. Loaded configuration may remain usable during an outage, while cached permissions become too old to justify access. Declare acceptable policy age, the evidence establishing that age, and which operations fail closed after expiry. Complete mediation—checking authority at every protected access—also applies during recovery. [66]

Retries can multiply load. If three nested layers each make four total attempts, one logical request can cause 64 calls at the deepest dependency. A retry budget bounds repeated attempts; it does not automatically bound monetary cost. Assign retry ownership, propagate the original deadline and distinguish permanent failures from recoverable ones. Failure classification and bounded retries covers implementation. [67]

A failure-triggered circuit breaker temporarily stops calls to a repeatedly failing dependency, then permits a recovery probe. That differs from a resource-threshold breaker limiting outstanding work. Envoy documents the latter, including distributed-limit races. Neither mechanism alone provides an exact global tenant spending reservation. [68][57]

Fallback must preserve required capabilities, processing permissions and application acceptance. Degradation may return less complete or older information only when the application permits it and exposes the limitation. If no permitted route or adequate capacity remains, explicit failure is a valid outcome. Availability cannot justify silently changing the task's evidence requirements. [36][7]

Release restoration also intersects data withdrawal. If the old assistant pairs encoder A with an index containing withdrawn material, restoring that bundle revives prohibited use. Substituting an index built for encoder B is not justified by equal dimensions. Recovery needs permitted, compatible dependencies and renewed acceptance checks. Serving restoration and interrupted requests covers backend re-entry. [32][69]

Supported workflows and adoption evidence

A golden path is a supported default workflow combining reusable capabilities, documentation and integrated controls. APIs, client libraries, templates and a portal can expose the same contract. Documented extension routes should preserve authority and operating limits while allowing application-specific choices. A portal is one interface to the platform, not the complete service. [1]

A bounded pilot should measure time to first supported deployment, repeated successful use, support effort per participating team and duplicated operating tasks removed. Include failed attempts and identify teams outside telemetry coverage. Combine instrumented use with structured feedback. These are proposed pilot measures; mandatory adoption and endpoint counts do not establish saved work. [71]

Compare like workloads and record concurrent changes before attributing improvement to the platform. Increased support demand can reveal confusing self-service rather than successful adoption. A team serving established needs can offer a service; uncertain requirements may need collaboration, while adoption difficulties call for facilitation. These are interaction choices, not a mandatory organizational redesign. [72][73]

Operating ownership and service retirement

Continuing ownership includes support scope, incident command, escalation, changes and retirement. Managed services transfer specified infrastructure duties; application authorization and appropriate data use remain organizational responsibilities. Operating inference directly adds capacity, runtime and patching work. Compare both arrangements against the same service promises rather than assuming either arrangement removes accountability. [58]

Supported versions need a lifecycle. Locate consumers, announce retirement, qualify replacements and verify that traffic migrated. Anthropic documents that retired-model requests fail and recommends task-specific replacement tests; usage exports help locate remaining consumers, while partner hosting schedules can differ. A successful new deployment does not prove that old consumers stopped calling the retired dependency. [75]

Tenant offboarding requires separate outcomes for new access, active jobs, credentials, artifacts, telemetry and unresolved external copies. Revocation can propagate after acknowledgement and cannot undo completed effects. Preserve an accountable owner until outstanding work and retained-data obligations are resolved; deleting an account record is only one operation. [38][15]

Expose usage attribution and allocation rules even when a supplier operates the backend. Task, attempt and shared-cost attribution explains the accounting. For the assistant, assign owners to timely authorized answers and compatible retrieval. For classification, assign owners to bounded waiting and durable accepted outcomes. Shared infrastructure supports both; their application promises remain distinct. [62][74]

Open questions

1. 01

   Policy freshness during control-plane outages remains a deployment-specific tradeoff. Continuing useful work can conflict with observing recent revocations. Progress requires an explicit maximum policy age and failure exercises showing which protected operations stop after expiry, including queued and already active work. [65][66]

2. 02

   End-to-end withdrawal remains difficult when lineage crosses services, external processors and restored backups. Acknowledgements can arrive before fulfillment, and some destinations remain unobservable. Progress would be a destination-complete exercise that keeps unresolved copies visible and prevents withdrawn content from becoming usable after restoration. [34][37]

3. 03

   Exact shared-budget enforcement across concurrent gateways remains harder than local throttling. Replicas can admit work before observing each other's consumption, while operations have different prices. Progress requires tested reservation and settlement behavior under races, failures and late usage reports, with an explicit bound on possible excess. [57][67]

4. 04

   Demonstrating platform value across unlike teams remains an attribution problem. Adoption telemetry may omit alternative implementations, and workflow changes can confound before-and-after results. Progress would combine declared coverage, comparable workload groups, repeated successful use and total support effort, including unsuccessful onboarding. [71][72]