Local and On-Device AI

Deployment goals and information boundaries

On-device means execution on the user's endpoint. Here, local also includes a stated nearby computing environment. A nearby server remains another machine, with its own administrator and communication boundary; geographical proximity does not establish ownership. [2]

Inference applies an already trained model without updating its learned parameters. The request-serving path explains that operation. Application location and inference location can differ: a function called from a laptop may execute on a remote GPU, as demonstrated by Runpod's decorated-function workflow. [3][4]

Local cost includes allocated hardware depreciation, energy, maintenance, software, and staff time. Count depreciation or the full purchase in a period model, not both. Low utilization spreads fixed costs over fewer completed tasks. Remote comparison must include service charges and retained integration costs; electricity alone does not represent full local ownership cost. [11]

A trust boundary separates assumptions about control or authority. Local processing changes one boundary; privacy also depends on collection, access, disclosure, and retention. An information inventory should trace prompts, outputs, local records, backups, synchronization, and diagnostic traffic. A model service's isolation or non-retention behavior does not describe everything its calling application does. [9][12]

Derived information can remain sensitive. An embedding is a numerical representation of input; researchers have reconstructed text from embeddings under specified model-access conditions. Local conversion into vectors therefore does not establish safe disclosure. Minimization limits what is collected and transferred rather than assuming transformation removes identification risk. [13]

The host must also be trusted: operating-system policy can permit inspection of another process's memory. Process separation alone does not protect prompts from a compromised host. Downloaded model files create a separate artifact-exposure surface. Finally, an air gap excludes physical connections and automated logical transfers; temporary disconnection or a loopback endpoint does not establish one. [14][15][16]

Workloads and device acceptance conditions

A workload contract states useful results, input and output shapes, arrivals, concurrent work, deadlines, and session conditions. A service objective is a declared acceptance target. Model size cannot substitute for these requirements: a narrow task may benefit more from specialization than from broad generation capabilities. [5][17]

Tokens are text units represented by vocabulary IDs, not necessarily words; Tokenization explains their boundaries. For summarization, record the joint distribution of complete input and output lengths. Long inputs paired with long outputs create a different device workload from either length in isolation, as described in workload shape and service objectives. [18][19]

The following requirements define two example applications; they are acceptance targets, not published device results. Summarization produces a bounded document result. Dictation continuously converts incoming speech into text, and early partial text can change before finalization. [20][21]

Time to first token, or TTFT, measures request submission to receipt of the first generated response. Completion latency ends at the final response. Neither necessarily measures first useful content. A p95 is a percentile of specified observations, not an average; report deadline misses as well. Bundled streaming responses do not expose every internal token-emission time. [24]

Fast partial transcription can coexist with late or unstable final text. Conversely, a quick summary opening can precede an incomplete answer. Separate these milestones using the latency boundaries. Task effectiveness needs its own evidence: MobileAIBench separates task evaluation from mobile resource measurements, and its summarization overlap scores do not by themselves establish factual fidelity. [20][22]

Peak memory and shared device capacity

Download size measures transferred files; installed storage measures persistent assets. Resident memory is memory currently held in RAM by a process. Peak running demand includes more than weights: request state, temporary execution storage, media, runtime, and application allocations coexist. The operating system and competing applications need capacity outside that budget. [28]

Weights are learned numerical parameters. Activations are intermediate numerical results; scratch space is temporary execution storage. A language model's KV cache retains context-dependent attention keys and values, not tokens themselves or durable conversation records. KV state and the resident memory budget explains why that request-dependent allocation must remain separate from shared weights. [3][29][30]

Assume the application has a 2,200 MiB allocation allowance after reserving capacity for the system and competing work. MiB means 1,048,576 bytes. The ledger below supplies the plotted totals. Temporary loading allocations exceed the allowance even though every later phase fits; independently adding category maxima would instead overestimate simultaneous demand. [1][28]

A published fixture illustrates the file-versus-process distinction without supplying an allocation breakdown. Google's FunctionGemma Mobile Actions dynamic-int8 configuration reports a 288 MB model and 551 MB peak RSS, or resident set size. The test used a Samsung S25 Ultra CPU, LiteRT XNNPACK, four threads, 512 prefill tokens, and 32 decode tokens. RSS is not whole-device demand. [32]

Dynamic KV allocations grow with retained history; static caches reserve capacity in advance. Windowed architectures follow different retention rules. More cache slots do not extend a model's supported sequence length. Limit complete input plus reserved output, using complete-request budgeting, and qualify concurrency separately because simultaneous requests can require additional state. [31][33]

Unified memory lets processors access a shared pool; MLX on Apple silicon avoids explicit array copies between separate CPU and GPU memory spaces. Count shared allocations once. Separate-memory systems require a budget for each pool and any duplicated buffers. Neither arrangement removes competition for capacity. Under memory pressure, a mobile operating system may terminate the application. [34][35][36]

Actual execution across processors

An inference runtime loads a model and executes its numerical operations. The CPU is a general-purpose processor; the GPU executes parallel workloads; an NPU, or neural processing unit, specializes in neural-network operations. A backend or execution provider is the runtime component that executes supported work on a processor. [37][35][38]

Capacity determines whether allocations fit. Memory bandwidth determines how quickly operands move; arithmetic capacity determines how quickly operations execute. Dispatch and synchronization add overhead. Inference bottlenecks depend on workload shape and hardware. An advertised peak arithmetic rate omits these other limits and cannot predict complete application latency. [39][40]

Prompt processing and iterative generation can stress different resources. Processing many prompt positions together often offers more arithmetic reuse; small-batch decoding can depend heavily on repeatedly moving weights and retained state. These are workload tendencies, not universal diagnoses. The DGX Spark account similarly distinguishes enough memory to hold a model from enough bandwidth to run it responsively. [41][42]

An operator is a numerical operation in the model graph. ONNX Runtime asks execution providers which operations they support, then assigns graph regions accordingly. A successfully loaded model therefore does not establish that every operation reaches the intended accelerator. Preprocessing and postprocessing also contribute to the application path. [37][38]

Verify placement with execution traces, operator timings, and accelerator activity. Windows NPU tooling can identify processes submitting work; hardware presence alone cannot. A utilization percentage also cannot isolate useful computation from waiting or overhead. Detailed kernel diagnosis belongs in finding the limiting resource. [38][44]

Compatible models, artifacts, and runtimes

Quantization represents numerical values using a smaller set of possibilities, often with fewer bits. It can reduce weight storage and memory traffic, but quality and execution cost depend on the model, format, and supported operations. Weight-only quantization does not automatically shrink KV state, activations, or workspace. Quantization covers numerical formats and calibration. [45][46]

Select for the required task before selecting for size. FunctionGemma targets structured mobile function calls and is not intended as a direct dialogue model. Its reported task-specialization gain does not establish successful authorized actions. A fluent general-purpose model and a narrow action model solve different parts of an application contract. [32][47]

A model artifact is a deployable representation of model data. Conversion changes that representation for a target runtime. A hardware delegate assigns supported operations to specialized execution. Architecture, artifact format, and runtime are separate choices: llama.cpp, for example, uses GGUF files and supports several processor backends and quantization formats. [48][37]

Models sharing an architecture can require different message serialization. Incorrect role markers or duplicated special tokens can change behavior even when execution succeeds. Preserve the selected checkpoint's preprocessing contract; tokenizer and model compatibility explains this interface. [18]

Model-card license metadata helps locate terms; it is not the complete agreement. Download availability and a runtime's software license do not establish permission to distribute model weights. Review the exact intended use and artifact separately. Compatibility tables suit this decision because each row records a distinct requirement rather than a causal sequence. [15]

Installation and changing readiness

Provisioning makes required assets available before use. A model bundle includes weights, tokenizer or other preprocessing assets, and configuration. Installing an inference library does not provision that bundle. Foundry Local's demonstration downloaded models in advance to remove the model-fetch dependency during execution. [7][51]

A cold start begins without the relevant loaded execution state; it does not describe physical temperature. Warmup performs initialization or preliminary execution before measured work. Initialization, warmup, and steady inference are separate measurements. Keeping a model loaded can avoid repeated startup but retains memory; any energy advantage requires a full-session comparison. [52][53]

Readiness means the required assets and execution state are usable now. Firebase distinguishes unavailable, downloadable, downloading, and available models; warmup separately initializes execution. An inference request does not automatically download missing assets. Its configured execution mode determines whether local unavailability causes failure or permitted cloud fallback. [54]

Initialization caches are implementation-specific. Apple's Stable Diffusion implementation distinguishes first-load preparation from subsequent cached loads, with behavior depending on asset form and compute selection. An earlier successful run cannot establish that a changed configuration will reuse the same preparation state. [1]

For application-managed bundles, require complete downloads, compatibility checks, staged activation, interrupted-install cleanup, and recovery to a known working bundle. Reserve storage for temporary assets and any retained previous version. These are implementation requirements: authenticated files alone do not establish atomic installation or successful initialization. [55][7]

Artifact verification starts with trusted update metadata and checks authorized signatures, versions, expiration, hashes, and lengths. TUF illustrates why a signature alone does not establish freshness. Authentication also does not prove model safety. Model artifacts and inference exposure separates these controls from protecting user inputs during execution. [55]

Sustained energy and thermal limits

Power, measured in watts, is the rate of energy use; energy, measured in joules, accumulates over time. A thermal budget is sustainable heat dissipation under current conditions; thermal throttling reduces performance near thermal limits. [56][57]

Duty cycle is the fraction of time actively computing. Continuous dictation and occasional summaries impose different active and idle patterns. Compare processor choice, residency, and work frequency over the whole session while holding quality and deadlines fixed. Lower instantaneous power can still consume more energy per useful task if completion takes longer. [27][56]

Assume one valid counter increases by 24,000,000 microwatt-seconds over 12 seconds while four useful tasks finish. That is 24 joules, two watts on average, and six joules per task. The result describes the measured rail and interval, not automatically the entire device or the application's isolated consumption. [56]

Use matching rail identities and counter epochs, rejecting resets. Android's power observations are device-level and can include other applications. Missing rails are unavailable coverage, not zero consumption. Battery percentage, charge, and instantaneous current are not themselves energy. Record elapsed time, ambient conditions, charging, cooling, screen activity, and background work alongside useful completions. [27][56]

Track latency, completions, and thermal state throughout identical sustained work. Correlated slowdown need not be exclusively thermal. Execution warmup does not establish thermal equilibrium. [57][52]

Admission, cancellation, and application lifetime

Admission control decides whether new work may start within resource and policy limits. Backpressure slows producers when consumers cannot keep up. Graceful degradation reduces work while exposing a narrower result. Bound input size, concurrency, and pending work; a request count alone ignores differences in resource demand. [58]

A queue does not automatically provide backpressure. Pipecat's inspected FrameQueue inherits unbounded capacity; its interruption reset is separate from limiting incoming work. For dictation, accumulating audio faster than it is processed increases delay and pending storage. For summaries, obsolete requests should not consume an unlimited queue behind the current request. [59][26]

A reduced-work mode must retain its declared usefulness. An approved smaller summarizer may meet the same fidelity target; simply discarding audio samples may destroy recognition evidence. If no validated reduced mode meets the contract, defer within a bounded deadline or report unavailability. General capacity scheduling belongs in LLM Inference. [17][21]

Platform eligibility is independent of spare hardware capacity. ML Kit GenAI requires the top foreground application and applies request and battery-use quotas; a foreground service can still be blocked. These restrictions apply to those managed APIs, not every Android runtime. [60]

Cancellation must reach the computation that consumes resources. gRPC generally cannot interrupt application handlers automatically; handlers must cooperate. Ending a stream therefore does not establish stopped inference. Verify queued-work removal and buffer release separately, and distinguish request-owned buffers from intentionally resident model allocations. [61][31]

Android may reclaim an application process even while a thread is doing work. Persist task identity and committed progress separately from transient inference state, then recheck readiness when resuming. Harness Engineering covers durable execution; progress and interruption covers communicating what actually stopped, survived, or needs restarting. [36]

Complete offline capabilities

Offline inference executes a model without a network dependency during execution. A complete offline task additionally needs dependency closure: every prerequisite required for its promised outcome must be available. Audit assets, authentication, local documents and indexes, preprocessing, tools, license checks, save locations, telemetry, and updates. [7][12]

Local computation can support complete, bounded actions. A demonstrated laptop agent interpreted dice requests and called a local random-number tool. That example establishes a narrow local execution path; it does not establish that arbitrary agents, remote tools, or their initial installation work disconnected. [62]

Generated text, a locally saved result, queued synchronization, and confirmed remote completion are distinct outcomes. Reconnection can introduce conflicts. Keep pending work visible and preserve authorization at disclosure boundaries; offline operation creates no exception to permissions. [63][47]

An unconfirmed remote mutation remains uncertain. A timed-out operation may already have changed the receiving system. Retain its operation identifier and reconcile before repeating it, using enforced idempotency where supported. Uncertain effects and safe retries explains why timeout and cancellation are not evidence of reversal. [64]

Local-cloud routing and transfer outcomes

Routing chooses an execution location for a request or stage. Cloud fallback uses remote processing when the preferred local route cannot proceed. The decision must join task capability, current resources, connectivity, latency, transfer cost, and data permissions. Local failure alone supplies no authority to disclose information. [65][66]

Firebase's PREFER_ON_DEVICE permits fallback, whereas ONLY_ON_DEVICE fails when local execution cannot proceed. Unsupported request features can also cause permitted fallback; responses identify the execution source. These settings implement routing behavior, not user authorization. The application must independently enforce and communicate its transfer policy. [54]

A handoff contract records payload identity, destination, authorization scope and lifetime, retained copies, application/model version, request identity, and expected result. Validate the result before downstream use. Authorization to process one document revision does not automatically authorize transferring its replacement. [66][47]

Model-reported confidence is not automatically a reliable escalation signal. An acceptance threshold changes how much work remains local, but the error rate among accepted cases must be measured. Calibration and selective automation explains that evidence. General routing optimization belongs in AI Cost and Performance Engineering. [69]

For a corrected summary request, two checks remain independent: whether an earlier transfer was authorized and whether its reply still belongs in the current view. Track request and revision identity. A superseded reply can be excluded from display while remote cancellation remains incomplete; suppressing display does not reverse remote effects. [61][64]

Qualification under real operating conditions

A deployment is qualified for a stated configuration and operating envelope, not for a model name alone. Record artifact revision, preprocessing, numerical format, runtime/backend, processor placement, OS, workload lengths, concurrency, arrivals, repetitions, and measurement boundaries. Engine benchmarks must be followed by measurements inside the actual application. [19][52]

Select prompts, formats, and other configurations using validation cases, then freeze them before final held-out evaluation. Repeatedly tuning against the final test set makes reported results optimistic. Controlled comparisons explain this separation. Device feasibility and task correctness need to pass together. [70][71]

Report failures and deadline misses alongside latency distributions. A remote benchmark includes network and client behavior; an inadequate load generator can conceal serving capacity. Separate those effects before attributing a result to the model. A single successful request cannot qualify the workload's arrival pattern or sustained concurrency. [72][19]

Keep process state and thermal state independent in the test matrix: a newly loaded model may run on an already hot device. Missing observations remain untested coverage. Telemetry resource limits explain why instrumentation cannot observe everything. Requalify affected claims after changes to models, numerical formats, backends, platform behavior, or application dependencies. [27][70][60]

Open questions

1. 01

   Portable admission estimates remain difficult because runtime allocations and competing application demand change independently. Better estimates would prevent avoidable failures without rejecting useful work. Progress would mean predicting peak demand within stated error bounds across loading, request lengths, concurrency, and unseen device conditions. [29][34]

2. 02

   The best residency and duty-cycle policy for intermittent assistance remains workload-dependent. Keeping execution state ready may reduce delay while changing retained resources and session energy. Progress would compare policies over identical arrival traces, including idle periods, with equal quality and deadline criteria and clearly bounded energy coverage. [53][22]

3. 03

   Reliable bundle replacement during disconnection remains an integration problem. Authenticity checks cannot ensure compatible initialization, and retained assets consume limited storage. Progress would demonstrate interrupted-download recovery, complete-bundle activation, and restoration of a working version without mixing preprocessing and weight revisions. [55][18]

4. 04

   Useful local-to-cloud minimization remains difficult because derived representations may retain sensitive source information. Smaller payloads alone do not establish reduced disclosure. Progress would pair task-utility measurements with explicit reconstruction or identification attacks under a stated recipient-access model. [13]