Placement and purpose
Where the request runs
Ollama’s hosted example posts messages and model: "gemma4:31b" to https://ollama.com/api/chat with a bearer key. The service generates the reply in message.content. For local generation, select a downloaded model at http://localhost:11434/api/chat. A signed-in local server can forward messages to a cloud model. Ollama
Mike Chambers ran Llama 3.1 8B with Ollama on his laptop. When he asked for a roll of a twenty-sided die plus five, the model selected a dice tool. A Python function on the same laptop rolled ten, and the app displayed fifteen. To see the distinction between the model and the software that executes it, consider llama.cpp. The GGUF file supplies the model’s weights. The llama.cpp runtime loads the file and executes the model, whether the caller uses the command line or its server API.
| Placement | Weights and computation | Who receives the prompt | Needed to run |
|---|---|---|---|
| In-process | User device, inside the app | Embedded runtime on the device | Compatible local runtime and model assets |
| Same-machine server | On-device service process | Local service, e.g. over loopback HTTP | Running service and downloaded model |
| Nearby server | Separate computer on the local network | That computer’s model service | Provisioned server and working local link, which can survive an internet outage |
| Hosted service | Provider’s serving infrastructure | Named remote service | Reachable, available service and required credentials |
The choice turns on what the application needs to control. Cormac Brick of Google AI Edge distinguishes using Android’s system-managed Gemini Nano through AICore from deploying an app-specific model. Both can compute on the phone. In the first case the developer checks whether the system supplies the capability. In the second the developer chooses and delivers the model for the task. Google
For Roboflow’s Joseph Nelson, the starting point is often a machine or robot with limited compute and intermittent connectivity. “I need this model to run in a limited context,” he explains in Why Local, Why Now. A specialized vision model can be worth deploying because the machine needs a particular visual decision when the link goes down. Soumith Chintala’s home Mac mini serves a different purpose. Connected to the internet, it gives personal agents a computer under his control on which to work asynchronously.
Moving a request onto the device removes the remote model round trip and transfers the work and energy demand to the device. To compare expenses, divide the cost over a stated period by the number of tasks completed to the same quality and deadline. Include failed attempts in the costs. Allocate the workload’s share of hardware, energy, software and support on the local side, and include service charges and retained operating costs on the hosted side. A lightly used dedicated workstation spreads its fixed costs over fewer completions. AI Cost and Performance Engineering
Where inputs and outputs go
Consider a reply displayed on a phone. During generation, the local runtime holds the input and temporary numerical state. Tapping Save creates a history entry with a different lifetime. In the figure, follow the branch from that entry to backup. Android Auto Backup can include internal databases, depending on the app’s rules and the user’s settings. An exchange that needed no server to generate can therefore reach a server later. Backup and device-to-device transfer also have distinct controls. Android backup rules
A trust boundary separates parties or components with different authority to access information. To trace it, name the fields in each input buffer, temporary file, tool argument, saved output, history and diagnostic record. For each copy, record its recipient, purpose and retention, including any synchronization or export. A report containing a runtime version and error code discloses different information from a report containing the failed prompt. Confidentiality concerns access by unauthorized parties. Privacy also concerns how an authorized recipient uses and retains information about a person. What governance decides · Inventory flows and copies
Saving introduces a second lifetime
ExampleThe backup path begins at the persistent database, after local generation.
Read the diagram as text
- Prompt on phone. User-supplied text.
- Phone runtime. Processes the prompt using temporary model state.
- Reply on screen.
- Local history database. Contains the saved exchange.
- External backup account.
- External diagnostics collector. An optional recipient in this design.
- Prompt on phone → Phone runtime: Prompt text.
- Phone runtime → Reply on screen: Generated reply.
- Reply on screen → Local history database: Exchange if user selects Save.
- Local history database → External backup account: Eligible file if backup runs.
- Phone runtime → External diagnostics collector: Version and error code if export enabled.
Account for traffic in both directions. A model download brings weights in. The download request can send credentials out, and a loader can check remote file metadata even for a cached model. The Hugging Face Hub client has separate controls for offline access, telemetry and implicit authentication. On the serving side, Ollama binds to 127.0.0.1 by default. A different bind address or a proxy can admit network clients. On the machine itself, file permissions and process-inspection rules still matter. Linux’s tracing policy, for example, controls when one process can inspect another’s memory. Hub client controls · Ollama serving FAQ · Linux Yama
Capability and development
Task requirements and model choice
A wake-phrase detector has a small output vocabulary and a demanding schedule. It must listen for hours, seldom interrupt someone who said something else, and still catch the intended phrase. A stricter detection threshold trades fewer false activations for more missed phrases. An on-demand assistant can rest between requests but has to interpret a much wider range of inputs. Specify the input type, languages, largest input, required output, tolerable errors and deadline before comparing model sizes. The candidate to test is a particular model, runtime and device capable of delivering that output. Evals and Benchmarks
| Candidate | Intent exact match | Structure exact match |
|---|---|---|
| MobileLLM-350M | 65.3% | 48.8% |
| LLaMA-v2 7B | 62.8% | 50.9% |
The smaller model selected the intended operation more often. The larger one more often matched the required structure. To turn either prediction into an action, the application must also validate the arguments and check permission for the requested target. The paper’s separate phone measurements used 125M variants, so neither row supplies a phone latency result. Structured Outputs and Tool Calling
Open weights are trained numerical parameters made available under the publisher’s terms. For a candidate that passes the task test, follow the exact revision’s model card to its agreement. For example, the Llama 3.1 agreement sets conditions on redistribution and attribution and incorporates an acceptable-use policy. Shipping weights with an app and installing a separately licensed runtime require checking each artifact’s terms. Technical eligibility is another decision. The runtime must understand the model architecture and its particular file format. Model-card license metadata · Llama 3.1 agreement
Recognition, mobile runtimes and portable generation
Early local recognizers gained useful capability by narrowing what people could ask of them. Apple’s Print Recognizer shipped on Newton OS 2.0 MessagePad 120 units in December 1995. Hand printing gave it pen lifts that helped separate strokes. A neural classifier scored possible characters, and a search considered alternative stroke groupings and words, including words outside the dictionary. Microsoft’s contemporary PC speech recognizer restricted the command grammar and compressed its acoustic model. It ran in real time on a 486DX PC with 4 MB of system memory, at the cost of making users phrase commands within the grammar. Newton Print Recognizer · Microsoft’s 1995 recognizer
As mobile applications took on continuous listening and camera input, engineers had to budget storage and arithmetic separately. In 2015, Tara Sainath and Carolina Parada evaluated small keyword detectors by comparing missed detections at a fixed false-alarm rate under a multiplication budget. Google’s 2017 MobileNets work gave vision developers two practical controls over resource use, network width and input resolution. Reducing the number of stored parameters alone could leave much of the work per image intact. Small-footprint keyword spotting · MobileNets
The TensorFlow Lite developer preview of November 14, 2017 separated model development from execution on a phone. In the announcement’s architecture figure, follow the converted model file. It travels to the device, where a small interpreter executes the operations it supports. Delivery still mattered. In Kelvin Ma’s account of Google Photos, the team used a shared inference library but downloaded an editing model after installation to keep the app download small. Google renamed TensorFlow Lite to LiteRT in 2024 as the tooling expanded beyond models authored in TensorFlow. Original preview and architecture figure · Google Photos · LiteRT announcement
On a microcontroller, a model can fit in storage yet need more RAM than the device has for intermediate results. In 2020, MCUNet addressed this by designing the network and its execution engine together. The engine planned when operations could reuse a buffer, and the search used that memory plan to find networks the device could run. TensorFlow Lite Micro’s 2021 paper described a reusable interpreter for devices that might lack a filesystem or a general-purpose allocator. The application supplies an arena, a contiguous region of memory. During initialization the interpreter assigns space within it so later operations can reuse memory once earlier results are no longer needed. MCUNet · TensorFlow Lite Micro
Local language generation posed a distribution problem as well as a capacity problem. An early llama.cpp README records a four-bit, seven-billion-parameter LLaMA model generating on a MacBook CPU, with conversion tools alongside the executable. Justine Tunney combined that runtime with Cosmopolitan Libc so a user could receive the weights and executable together in one portable file. Mozilla announced the first llamafile release on November 29, 2023. A generator became easier to try on an existing laptop, while an always-listening detector still had good reason to specialize in a single phrase. Early llama.cpp snapshot · Mozilla’s release announcement · Hood and Tunney on llamafile · Mozilla
Execution and resource fit
Runtime and processor placement
A runtime can divide one model’s computation among processors on the same device. It works from a graph whose nodes are operators, such as multiplication or reshaping, and whose edges carry numerical arrays called tensors. A backend implements operators for a particular processor. A central processing unit (CPU) runs general-purpose instructions, a graphics processing unit (GPU) can perform many calculations in parallel, and a neural processing unit (NPU) specializes in neural-network arithmetic. Which nodes can run on each processor depends on the backend’s support for their operator versions, numerical types and input shapes with the installed operating system and driver. Open Neural Network Exchange (ONNX) Runtime architecture · Microsoft’s guide to observing NPU execution
ONNX Runtime asks its backends, called execution providers, which parts of the graph they can handle. For a ResNet image model, its mobile usability checker estimated that the CoreML provider could handle 120 of 122 nodes, split into two groups. The checker flags an unsupported Flatten operation and a restriction on inputs whose shape can vary. The count tells you how much of the graph is eligible. An execution trace on the device shows where the operations actually ran for a given input. Published checker output
Look at the middle operation in the partition figure. A CPU backend can execute an operation the accelerator cannot handle, but the CPU must first obtain its input. The next accelerator operation then waits for the CPU’s output. With separate memory pools, each handoff can require a copy. Shared memory can remove the copy while preserving the dependency. Even a cheap CPU operation can make the split slower than CPU-only execution. Compare both paths with matched inputs. Where no backend supports the operation, the configuration can fail outright. LiteRT GPU delegate guidance
Deployment also determines who supplies the backend. Native apps can embed it. In Emma Ning’s 2025 Foundry Local demonstration, a Microsoft service hosted models on the client computer using ONNX Runtime. Its catalog exposed hardware-specific variants. A Qualcomm NPU variant was absent on Ning’s machine, which lacked that NPU. In a browser, ONNX Runtime Web needs matching WebAssembly runtime binaries, model files and any external tensor data. Worker loading must satisfy the page’s origin and security rules, and WebGPU requires a secure context. ONNX Runtime Web deployment
Peak resident memory
A useful pair of numbers appears in Google’s FunctionGemma model card. The Mobile Actions dynamic-int8 variant occupied 288 MB as a model file and reached 551 MB peak resident set size during the device test. Resident set size (RSS) counts the process’s private and shared pages present in RAM. The test used a Samsung S25 Ultra CPU, LiteRT’s XNNPACK backend and four threads, with 512 prompt tokens, 32 generated tokens and a 1,024-token context configuration. Use the first number to plan storage. Use the second to understand process memory under those conditions. Subtracting them would not isolate a particular runtime buffer. FunctionGemma device measurements
Budget simultaneous demand in each physical pool. Include resident weights, intermediate model values called activations, runtime workspaces, app buffers and state private to each active request. Leave headroom for other applications and the operating system. Reused storage counts once while two live copies count twice. MLX on Apple silicon lets CPU and GPU operations address the same array in unified memory. For separate processor memories, charge actual copies to their respective pools. Across Android processes, proportional set size (PSS) attributes a fraction of a shared page to each process. Adding RSS values would count that page repeatedly. MLX unified memory · The capacity envelope
Request demand against usable RAM
ExampleA longer input can cross the same allowance with the same weights.
Short input
3.5 GiB required, leaving 0.5 GiB free.
- 1. Weights
- 2. Other live buffers
- 3. History cache
- 4. Usable allowance
Read coordinates and regions as data
X: 0–1 dimensionless; Y: 0–5 GiB, increasing up. Axes scaled independently; screen angles and distances are not comparable.
(0.2, 0); (0.8, 0); (0.8, 2); (0.2, 2)
(0.2, 2); (0.8, 2); (0.8, 3); (0.2, 3)
(0.2, 3); (0.8, 3); (0.8, 3.5); (0.2, 3.5)
(0.05, 4); (0.95, 4)
Weights 2 GiB: (0.5, 1)
Buffers 1 GiB: (0.5, 2.5)
Cache 0.5 GiB: (0.5, 3.25)
4 GiB allowance: (0.5, 4.25)
Long input
4.5 GiB required, exceeding the allowance by 0.5 GiB.
- 1. Weights
- 2. Other live buffers
- 3. History cache
- 4. Usable allowance
Read coordinates and regions as data
X: 0–1 dimensionless; Y: 0–5 GiB, increasing up. Axes scaled independently; screen angles and distances are not comparable.
(0.2, 0); (0.8, 0); (0.8, 2); (0.2, 2)
(0.2, 2); (0.8, 2); (0.8, 3); (0.2, 3)
(0.2, 3); (0.8, 3); (0.8, 4.5); (0.2, 4.5)
(0.05, 4); (0.95, 4)
Weights 2 GiB: (0.5, 1)
Buffers 1 GiB: (0.5, 2.5)
Cache 1.5 GiB: (0.5, 3.55)
0.5 GiB over: (0.5, 4.75)
Mapping a model file establishes a virtual address range. On Linux, accessing a mapped page can still require fetching it from storage. A mapping can therefore succeed before the application encounters expensive page faults. Request state adds a different pressure. A language model’s key-value cache retains attention data derived from earlier tokens. A growing cache uses more space as history lengthens. A static cache reserves its maximum shape up front, while a sliding-window layer can bound how much history it retains. Allocating more slots does not extend the sequence length supported by the model. Linux file mappings · The KV cache
In the illustrative memory comparison, both requests use 2 GiB of resident weights and 1 GiB of other live allocations. Only the input length changes. A full-history cache needing 0.5 GiB fits under the 4 GiB allowance. At 1.5 GiB of cache, demand reaches 4.5 GiB. The upper band shows the extra space the second request would need. For image inference, larger feature maps can increase activation storage instead. For concurrent requests, include every private cache that can be live at once before choosing how many requests to admit.
Precision and device fit
Quantization represents numerical values approximately using fewer distinct levels. On the memory plot, a weight-only change targets the bottom band. If the backend could halve its 2 GiB resident weight allocation while keeping all other allocations fixed, the longer request would need 3.5 GiB. To find the real saving, inspect the package and the execution path. Packed weights need metadata such as scales, and the backend may expand values into a wider representation for computation. Cache storage can use a separately configured precision. What quantization changes
A model can use lower precision in some layers and keep more precision where rounding would change a decision. This is mixed precision. In LinkedIn’s 360Brew ranker, the output head is the part of the model that scores candidates. The team found that lower precision there made nearly equal probabilities hard to distinguish, so they kept the head at 32-bit floating-point precision. When comparing variants for a device, run the same task cases on the same backend and inspect which decisions change. Measure peak memory, latency and energy separately. A backend that must convert packed weights before using them can save space with little gain in speed. 360Brew · Compression at the Edge · Shelby Heinecke on evaluating efficient models
Time to a usable result
Time the event the user is waiting for. A client can mark submission, request send, first response received, first useful content displayed and completion. For a text model, prefill processes the prompt and decode produces successive output tokens. A long prompt can delay response onset even when decoding is fast. In streaming measurements, one response chunk can contain several tokens. A user waiting for a proposed command may need the whole command before any of it is useful. Prefill and decode · Latency has boundaries
Watch the reply grow in Adrien Grondin’s offline iPhone demonstration. He reports about 40 generated tokens per second for his four-bit Gemma 4 configuration with MLX. That rate describes output once generation is underway. A timestamp at submission and another at first usable display would be needed to measure the initial wait. A prolonged run would answer a third question about sustained speed.
What occupies the submit-to-display interval
Example timingsHalving accelerator execution saves 15 ms of the 120 ms wait.
Read the diagram as text
- Submit to usable display. 0 to 120 ms; duration 120 ms.
- Wait for admission. 0 to 20 ms; duration 20 ms. Parent: Submit to usable display.
- Prepare input. 20 to 50 ms; duration 30 ms. Parent: Submit to usable display.
- Copy input to accelerator. 50 to 65 ms; duration 15 ms. Parent: Submit to usable display.
- Compute model output. 65 to 95 ms; duration 30 ms. Parent: Submit to usable display.
- Return output. 95 to 105 ms; duration 10 ms. Parent: Submit to usable display.
- Interpret and display. 105 to 120 ms; duration 15 ms. Parent: Submit to usable display.
Trace queueing and preparation as well as processor time. In the example timeline, accelerating a 30 ms model computation by a factor of two removes 15 ms from a 120 ms wait. The copies and display work still take time. Even inside the model, peak arithmetic throughput is relevant only when arithmetic limits progress. If the processor spends most of its time fetching weights for one user’s next token, memory bandwidth, measured in bytes moved per second, can limit the output pace. Roofline performance model
For continuous media, compare service rate with the arriving stream. Speech recognition’s real-time factor is processing time divided by audio duration. Eight seconds to process ten seconds of audio gives 0.8. A warm test on complete audio buffers can achieve that rate while live dictation still displays words late or revises them for too long. Measure partial and final text separately. For video, count frames processed before their deadline and frames skipped. Bound input size, queue length and concurrency according to those deadlines. A frame processed after it can influence the next decision consumes resources without serving its purpose. Speech measurement boundary · Scheduling and admission
Delivery and lifetime
Installing a compatible model package
Package the capability that passed the test. A manifest identifies one release’s weights, architecture, input processor, output conventions, compatible runtime and distribution terms. A chat model can load with the wrong vocabulary or message template and still produce text. The vocabulary assigns token identifiers, while the template inserts the boundaries that tell the model who spoke. Changing either can change the meaning of its input. An image classifier likewise depends on expected pixel normalization and a matching list of output labels. Run a known input through preparation, execution and output interpretation to catch a mixed package. Tokenization · LiteRT model metadata
Bundling delivers assets at app installation. An on-demand download can supply only the features or languages the user needs, but arriving bytes may still need preparation. In Apple’s Core ML workflow, the downloaded .mlmodel is compiled into .mlmodelc before the app creates a usable model. The compiled output starts in temporary storage. Moving it to persistent storage allows reuse at the next launch. Allow disk space for the currently active release, the incoming release and compilation output during the switch. Apple’s download and compilation guide
Activating a complete package
ExampleA validated replacement can exist on disk before any new session uses it.
Transfer B outside A’s directory. An interruption leaves A selected.
Read the diagram as text
- Release A. Weights, input processor and configuration in directory A.
- Release B. Separate destination for the new compatible set.
- Active record selects A.
- Staged transfer.
- B passed validation. Complete, authenticated and tested together.
- Active record selects B.
- Retained for existing readers.
- Active record selects A → Release A: New sessions open.
- Staged transfer → Release B: Writes replacement files.
- B passed validation → Release B: Checks apply to this set.
- Active record selects B → Release B: New sessions open.
- Retained for existing readers → Release A: Keeps old files accessible.
- Stage the replacement. Transfer B outside A’s directory. An interruption leaves A selected. Active: Release A, Release B, Active record selects A, Staged transfer. New: Release A, Release B, Active record selects A, Staged transfer.
- Verify B as a set. New sessions still open A while B becomes eligible for use. Active: Release A, Release B, Active record selects A, B passed validation. New: B passed validation.
- Commit the active record. Select B for new sessions. Readers that opened A can finish with A. Active: Release A, Release B, B passed validation, Active record selects B, Retained for existing readers. New: Active record selects B, Retained for existing readers.
Before activating a download, verify who authorized its contents. The Update Framework starts from trusted keys, authenticates release metadata and checks target file lengths and hashes. Metadata versions and expiry checks also help detect rollback, inconsistent sets and stale metadata. Then verify compatibility and behavior on the device. A correctly signed package can still require a runtime feature the installed version lacks. The Update Framework specification · AI Security
The installation figure shows a design for surviving an interrupted update. Each release has its own directory. A session reads one active-version record and resolves all its files from that release. In the middle step B is complete and tested, yet new sessions still open A. Replacing the small record atomically selects B. Android’s AtomicFile can commit that one file. The installer must coordinate writers, finish and persist the referenced files first, and keep A while existing readers need it. A later recovery can select A only while that complete release remains compatible and authorized. AtomicFile contract
Cold starts, readiness and residency
In the Foundry Local demonstration, Ning used foundry cache list to inspect models she had downloaded before presenting. This removed the need to acquire those models during the demonstration. A disk inventory answers a different question from whether the next call can run immediately. Loading weights, creating the execution session, compiling for a target and warming up can still precede a first result. Eager loading moves that delay before demand. Lazy loading avoids the work entirely when the feature goes unused. Keeping a session resident saves initialization on subsequent calls and occupies memory between them.
| Entry condition | What survives | What the next request needs |
|---|---|---|
| First installation | App and bundled assets | Acquire missing assets, verify and prepare the release, then initialize. |
| Installed cold start | Complete release and valid persistent compilation output | Load or rebuild the session locally and warm it as required. |
| Warm request | Compatible resident session | Prepare the new input and execute. |
| Mandatory asset purged | An incomplete set | Restore the missing file and validate readiness before serving. |
Recovery depends on which resource disappeared. Android can terminate an application under memory pressure. Chrome can freeze a tab or discard it without a final cleanup callback. Save recoverable task inputs and progress before either event. Reopening the client can reconstruct a session from surviving assets even though the interrupted computation is gone. Test session creation again after accelerator loss. Chrome’s managed models can also be purged under disk pressure or changed eligibility, in which case the app has to initiate another download. Chrome page lifecycle · Built-in model management
Cancellation also has a scope. Chrome’s Prompt API accepts an abort signal for one prompt. Destroying the session aborts its work and releases session resources, while the lifetime of shared weights is a separate matter. In ONNX Runtime’s Java API, setTerminate(true) targets every incomplete call using the same RunOptions instance. Give independent requests separate cancellation scopes and test that cancelling one allows the other to finish. Chrome session management · ONNX RunOptions · Design Engineering and AI Interfaces
Sustained operation
Energy, heat and sustained work
Power is the rate of energy use. For example, a task averaging 2 watts over 3 seconds uses 6 joules. Finishing the same task in 1 second at 3 watts uses 3 joules. To compare battery demands, measure energy over the intended period of use and divide by useful completions. Include sensing, memory traffic and idle intervals that the feature requires. A thermal budget is the heat load the device can sustain under specified conditions. When heat accumulates, the device can reduce performance to control its temperature. This throttling changes both the time per task and the amount of work completed.
Apple’s 2017 iPhone wake detector illustrates a way to change the duty cycle, the fraction of time a component works. A low-power auxiliary processor checked audio continuously. Only a strong candidate woke the main processor for a more expensive check. Follow the weak-detection branch in the figure. The phone keeps listening while avoiding the larger computation for that candidate. Tightening the first threshold can save work but can also discard a genuine activation before the second detector sees it. Apple’s Hey Siri engineering account
Which detections wake the main processor
Only candidates that pass the low-power check incur the larger local check.
Read the diagram as text
- Microphone.
- Auxiliary detector.
- End candidate check.
- Main processor checks candidate.
- Reject activation.
- Accepted local activation.
- Siri server.
- Microphone → Auxiliary detector: Audio samples.
- Auxiliary detector → End candidate check: Weak detection.
- Auxiliary detector → Main processor checks candidate: Strong detection triggers wake.
- Main processor checks candidate → Reject activation: Second check fails.
- Main processor checks candidate → Accepted local activation: Second check passes.
- Accepted local activation → Siri server: Accepted audio if sent.
Repeated requests can reveal a slowdown that a single request misses. The MELT researchers ran four-bit Zephyr-3B through 50 consecutive prompts on an iPhone 14 Pro and repeated the experiment three times. They observed declining throughput and proposed changes in the phone’s power management and clock frequency as a possible cause. In a chat app, a growing conversation can also slow generation because each new token can require reading more history. To distinguish these effects, save a short history and a long one. Replay each on the same phone both cool and warm, keeping the model loaded and the backend, output length and concurrency fixed. Compare temperatures using the same history, then compare histories at the same temperature. MELT study
Record ambient temperature, charging and battery state, competing applications, thermal status and timely completions over the same observation window. State which power rails were measured. Android’s Power Profiler observes device activity, so another app can contribute to those readings. If the workload is unsustainable, choose the concession the task allows. Sampling less often can miss a brief event. An output limit may leave a response incomplete. Deferral can work for a batch with a later deadline, provided the runtime is available then. ML Kit GenAI requires the top foreground app. Plugging in the phone while another app occupies the screen does not make that managed API available to a background job. Android Power Profiler · ML Kit GenAI resource contract
Connectivity and cooperation
What works offline
Offline inference needs no live network connection to execute the model. An offline feature must also acquire its input, interpret the result and complete its promised action with locally available dependencies. Test a fresh installation separately from a provisioned cold start. For the latter, stop the model-hosting process as well as the client, disconnect, then request the feature. Transformers can load a previously downloaded model with local_files_only=True. A browser deployment may also need a cached runtime binary and external tensor files. Neither loader supplies an unavailable tool or refreshes an expired entitlement. Transformers offline loading
In Brick’s LiteRT-LM workshop, one skill could run local JavaScript, while a music skill required a key and a web service. A prototype Wikipedia archiver fetched articles and indexed them ahead of time. After disconnection it could search a saved article but could neither supply a missing article nor discover a later edit. Give users the collection date and scope when the available result depends on them. The same limit applies to authority. A disconnected app can enforce a local expiry, but it cannot immediately learn that a remote party revoked access. Search and Retrieval · Enforce current authority
| Condition | Outcome the user can rely on | Behavior on reconnection |
|---|---|---|
| Complete installation, valid local authority and local action | Complete the task and save its result locally. | No transfer is needed. |
| Missing mandatory asset or expired required authority | Identify the unavailable capability. Preserve any permitted draft. | Acquire the asset or re-establish authority before proceeding. |
| Older reference collection | Offer a dated result if the task permits stale data. | Refresh according to the collection’s policy. |
| Remote action requested while offline | Offer a draft, or a pending send if the later transfer is explicitly authorized. | Check that authorization still applies before sending. |
| Connection lost after transmitting a write | Mark the effect uncertain and retain its operation identifier. | Reconcile with the recipient before retrying. The original attempt may have succeeded. |
Authorized cloud handoffs
A workflow can deliver a useful local result before offering a remote operation. Google’s 2019 Recorder implementation transcribed speech, indexed words and retained their timestamps on the device. Searching for a word could jump to the corresponding audio without internet access. Today’s Recorder help treats summarizing a saved transcript and re-transcribing audio as separate operations with their own eligibility and cloud-processing conditions. Permission and connectivity requirements therefore attach to the particular operation. Recorder engineering account · Summary and re-transcription guidance
Consider a proposed notebook that offers to summarize only a selected passage. It can show the passage, identify the summary service and explain that service’s use and retention of the text. Check permission for that transfer before invoking the service. In the diagram, the transcript is already usable on every branch. Denial keeps the selection local. With approval but no connection, the notebook can report that nothing was sent. When approved and connected, it sends the selected text, receives the summary and identifies its remote origin. Microphone permission covers capture. The outbound passage needs its own authorization. Media Capture and Streams · Approve actual service paths
A remote summary of selected text
ExampleOnly the selected passage has a permitted route to the summary service.
Read the diagram as text
- Local audio.
- Local timestamped transcript. An independently useful result.
- Check proposed send. Match selected text, recipient, purpose and retention to approval, then check the connection.
- Keep selection private.
- Report passage unsent. No automatic upload is scheduled.
- Approved remote summary service. Only the recipient identified in the permission check.
- Summary displayed on device.
- Local audio → Local timestamped transcript: Local recognition yields words and times.
- Local timestamped transcript → Check proposed send: Selected passage.
- Check proposed send → Keep selection private: Stop if approval absent.
- Check proposed send → Report passage unsent: Stop if approved but offline.
- Check proposed send → Approved remote summary service: Selected text if approved and reachable.
- Approved remote summary service → Summary displayed on device: Returned summary text.
Changing the representation can reduce what is sent while preserving sensitive information. Selected words can name a person. A numerical embedding can preserve clues to the original text. Researchers with target embeddings, a trained inversion model and query access to the embedding model recovered names from embedded clinical notes. A different placement technique, split inference, divides the numerical operations of one model between machines. The 2017 Neurosurgeon study explored sending intermediate arrays at such a split because network conditions could change whether offloading saved time or device energy. Each design needs an inventory of what its recipient actually receives. Embedding inversion study · Embeddings and Representation Learning · Distributed Training and Inference
A third design routes the whole request according to local capability. In Florina Muntenescu and Oli Gaymond’s Gemini Nano account, cloud execution expands coverage to devices without the local model. Firebase AI Logic distinguishes unavailable, downloadable, downloading and available states. Submitting a request does not download the model. ONLY_ON_DEVICE fails when local execution cannot proceed. PREFER_ON_DEVICE allows fallback, including for unsupported features. Check the permitted recipient before invoking that mode and show the execution location reported in the response. Resource pressure can make a route infeasible, but only the application’s data policy can authorize another recipient. Google DeepMind · Firebase hybrid inference · Model Routing and LLM Gateways
Deployment qualification
Evidence for device support
Qualify a versioned capability under named conditions. For each device and OS group, record the package revision, input processor, runtime, actual backend, admitted input and output sizes, concurrency and entry state. Include the lowest-memory target. Set task criteria and deadlines first, use development cases to choose a configuration, then assess held-out cases. A faster run that omitted a required part of the answer failed the task. Evals and Benchmarks · Benchmark the workload
| Promise | Exercise | Passing observation | Decision if unmet |
|---|---|---|---|
| Task usefulness | Held-out inputs across supported languages and sizes | Outputs meet the error and completeness criteria. | Narrow the task or change the model. |
| Execution target | Actual device, drivers and input shapes | Operator trace confirms placement and results pass. | Qualify another backend separately. |
| Installation and recovery | Interrupt an update, kill the host, restart offline, remove a required file | A complete supported release runs when promised. Missing dependencies are identified. | Repair provisioning or restrict the offline promise. |
| Memory fit | Largest inputs and maximum admitted overlap | Peak in every pool fits its usable allowance. | Limit request size, residency or concurrency. |
| Responsiveness | Cold and warm requests at the intended arrival rate | First useful display and completion meet their respective deadlines. | Trace the slow stage or restrict the mode. |
| Sustained work | Repeated tasks under stated power, thermal and background conditions | Timely useful completions stay within the energy allowance. | Reduce duty cycle or defer eligible work. |
| Data boundaries | Deny a transfer, disconnect mid-request, reconnect and exercise backup | Observed recipients, fields and deferred effects match the policy. | Disable the unqualified path. |
Keep the measurement window with the result. MLPerf Tiny’s 2021 procedure paired embedded tasks with quality targets and reported energy per inference in a window that excluded preprocessing and postprocessing. MobileAIBench evaluated task quality separately from its iPhone measurements of resource use and latency. A fast recognizer inside either kind of test would still need a measurement from microphone input to finalized text to qualify live dictation. MLPerf Tiny · MobileAIBench
Network evidence also has a boundary. A browser service worker can answer a page request from cache or make a fetch of its own. Disabling the worker changes the application being tested. Page observations alone cannot account for a native service or an OS backup. Record the route, version, stage and duration needed to diagnose failures while leaving user content out of routine diagnostics. A defensible release decision for the notebook would be to enable foreground transcription on configurations that passed, preserve the transcript when summarization is unavailable, and offer transfer only on a separately authorized and tested summary path. Service-worker observations · Observability
Frequently asked questions
Can an API call run entirely on my laptop?
Yes. Select a downloaded model on the laptop’s Ollama server.
Can a downloaded model start offline?
Yes, if the cold session can initialize using local files. Test with the hosting process stopped and the network disconnected. Include the runtime, tokenizer or media processor, and any external tensor files or browser runtime binaries the chosen package needs.
How soon will a streaming answer be useful?
Measure from submission to the first displayed content that satisfies the immediate need. A token rate measures output pace, while a first-response timestamp measures arrival at the client. Both can miss a wait for the app to assemble or display a usable result.
What happens when the local model is unavailable?
A local-only feature can report the missing capability or offer a qualified local alternative. A local-preferred feature can use a reachable cloud service if the payload, purpose and recipient are authorized. Check that permission before a fallback-capable request and report where it ran.
Open questions
How accurately can short tests predict useful work through an hour of mobile use? History length, competing applications and heat can all change output rate. Stock-device studies aligning timely completions, memory, energy and thermal state under controlled conditions would help teams set sustainable input and duty-cycle limits.
How long can a user rely on a model prepared before going offline? The platform can remove a session or purge an asset, and the two losses have very different recovery requirements. A verifiable retention interval or timely notice of impending loss would make advance provisioning more dependable.
Which representation discloses the least information while preserving a remote stage’s usefulness? Shortening a passage or sending a vector changes what the service can compute and what it may reconstruct. Matched task tests and reconstruction tests under the actual recipient’s access assumptions would give teams a basis for choosing a handoff.





































































































































