Contents
  1. Placement and purpose
    1. Where the request runs
    2. Where inputs and outputs go
  2. Capability and development
    1. Task requirements and model choice
    2. Recognition, mobile runtimes and portable generation
  3. Execution and resource fit
    1. Runtime and processor placement
    2. Peak resident memory
    3. Precision and device fit
    4. Time to a usable result
  4. Delivery and lifetime
    1. Installing a compatible model package
    2. Cold starts, readiness and residency
  5. Sustained operation
    1. Energy, heat and sustained work
  6. Connectivity and cooperation
    1. What works offline
    2. Authorized cloud handoffs
  7. Deployment qualification
    1. Evidence for device support
    2. Frequently asked questions
      1. Can an API call run entirely on my laptop?
      2. Can a downloaded model start offline?
      3. How soon will a streaming answer be useful?
      4. What happens when the local model is unavailable?
  8. Check understanding
  9. Open questions
  10. Selected talks
  11. References
  12. Talk library
← All topics

Local inference

Local inference is the execution of a trained model on the user’s device or on a nearby computer. On-device inference is the narrower case in which the device itself runs the model. In chat, the machine doing inference uses the model’s learned parameters, called weights, to compute each new piece of text, or token. Ollama’s chat API supports both local and hosted models.

Placement and purpose

Where the request runs

Ollama’s hosted example posts messages and model: "gemma4:31b" to https://ollama.com/api/chat with a bearer key. The service generates the reply in message.content. For local generation, select a downloaded model at http://localhost:11434/api/chat. A signed-in local server can forward messages to a cloud model. Ollama

Mike Chambers ran Llama 3.1 8B with Ollama on his laptop. When he asked for a roll of a twenty-sided die plus five, the model selected a dice tool. A Python function on the same laptop rolled ten, and the app displayed fifteen. To see the distinction between the model and the software that executes it, consider llama.cpp. The GGUF file supplies the model’s weights. The llama.cpp runtime loads the file and executes the model, whether the caller uses the command line or its server API.

Follow the same prompt across these four placements. The first two differ in process boundaries. The last two introduce another machine. A nearby server can belong to a different operator even when it sits in the same room. NIST’s fog computing conceptual model
PlacementWeights and computationWho receives the promptNeeded to run
In-processUser device, inside the appEmbedded runtime on the deviceCompatible local runtime and model assets
Same-machine serverOn-device service processLocal service, e.g. over loopback HTTPRunning service and downloaded model
Nearby serverSeparate computer on the local networkThat computer’s model serviceProvisioned server and working local link, which can survive an internet outage
Hosted serviceProvider’s serving infrastructureNamed remote serviceReachable, available service and required credentials

The choice turns on what the application needs to control. Cormac Brick of Google AI Edge distinguishes using Android’s system-managed Gemini Nano through AICore from deploying an app-specific model. Both can compute on the phone. In the first case the developer checks whether the system supplies the capability. In the second the developer chooses and delivers the model for the task. Google

For Roboflow’s Joseph Nelson, the starting point is often a machine or robot with limited compute and intermittent connectivity. “I need this model to run in a limited context,” he explains in Why Local, Why Now. A specialized vision model can be worth deploying because the machine needs a particular visual decision when the link goes down. Soumith Chintala’s home Mac mini serves a different purpose. Connected to the internet, it gives personal agents a computer under his control on which to work asynchronously.

Moving a request onto the device removes the remote model round trip and transfers the work and energy demand to the device. To compare expenses, divide the cost over a stated period by the number of tasks completed to the same quality and deadline. Include failed attempts in the costs. Allocate the workload’s share of hardware, energy, software and support on the local side, and include service charges and retained operating costs on the hosted side. A lightly used dedicated workstation spreads its fixed costs over fewer completions. AI Cost and Performance Engineering

Where inputs and outputs go

Consider a reply displayed on a phone. During generation, the local runtime holds the input and temporary numerical state. Tapping Save creates a history entry with a different lifetime. In the figure, follow the branch from that entry to backup. Android Auto Backup can include internal databases, depending on the app’s rules and the user’s settings. An exchange that needed no server to generate can therefore reach a server later. Backup and device-to-device transfer also have distinct controls. Android backup rules

A trust boundary separates parties or components with different authority to access information. To trace it, name the fields in each input buffer, temporary file, tool argument, saved output, history and diagnostic record. For each copy, record its recipient, purpose and retention, including any synchronization or export. A report containing a runtime version and error code discloses different information from a report containing the failed prompt. Confidentiality concerns access by unauthorized parties. Privacy also concerns how an authorized recipient uses and retains information about a person. What governance decides · Inventory flows and copies

Saving introduces a second lifetime

Example

The backup path begins at the persistent database, after local generation.

Prompt on phoneUser-supplied text.
Phone runtimeProcesses the prompt using temporary model state.
Reply on screen
Local history databaseContains the saved exchange.
External backup account
External diagnostics collectorAn optional recipient in this design.
The example app displays a local reply. Saving creates a history entry that may enter Android backup under applicable rules and settings. An optional diagnostic export carries only the two operational fields shown.
Read the diagram as text
  • Prompt on phone. User-supplied text.
  • Phone runtime. Processes the prompt using temporary model state.
  • Reply on screen.
  • Local history database. Contains the saved exchange.
  • External backup account.
  • External diagnostics collector. An optional recipient in this design.
  • Prompt on phone → Phone runtime: Prompt text.
  • Phone runtime → Reply on screen: Generated reply.
  • Reply on screen → Local history database: Exchange if user selects Save.
  • Local history database → External backup account: Eligible file if backup runs.
  • Phone runtime → External diagnostics collector: Version and error code if export enabled.

Account for traffic in both directions. A model download brings weights in. The download request can send credentials out, and a loader can check remote file metadata even for a cached model. The Hugging Face Hub client has separate controls for offline access, telemetry and implicit authentication. On the serving side, Ollama binds to 127.0.0.1 by default. A different bind address or a proxy can admit network clients. On the machine itself, file permissions and process-inspection rules still matter. Linux’s tracing policy, for example, controls when one process can inspect another’s memory. Hub client controls · Ollama serving FAQ · Linux Yama

Capability and development

Task requirements and model choice

A wake-phrase detector has a small output vocabulary and a demanding schedule. It must listen for hours, seldom interrupt someone who said something else, and still catch the intended phrase. A stricter detection threshold trades fewer false activations for more missed phrases. An on-demand assistant can rest between requests but has to interpret a much wider range of inputs. Specify the input type, languages, largest input, required output, tolerable errors and deadline before comparing model sizes. The candidate to test is a particular model, runtime and device capable of delivering that output. Evals and Benchmarks

Even for a single task, the winning candidate can depend on what the caller needs. In Meta’s MobileLLM study, the API-calling experiment used 5,000 generated examples for training and 2,500 for testing. One score checked the intended operation. The other checked the call structure.
CandidateIntent exact matchStructure exact match
MobileLLM-350M65.3%48.8%
LLaMA-v2 7B62.8%50.9%

The smaller model selected the intended operation more often. The larger one more often matched the required structure. To turn either prediction into an action, the application must also validate the arguments and check permission for the requested target. The paper’s separate phone measurements used 125M variants, so neither row supplies a phone latency result. Structured Outputs and Tool Calling

Open weights are trained numerical parameters made available under the publisher’s terms. For a candidate that passes the task test, follow the exact revision’s model card to its agreement. For example, the Llama 3.1 agreement sets conditions on redistribution and attribution and incorporates an acceptable-use policy. Shipping weights with an app and installing a separately licensed runtime require checking each artifact’s terms. Technical eligibility is another decision. The runtime must understand the model architecture and its particular file format. Model-card license metadata · Llama 3.1 agreement

Recognition, mobile runtimes and portable generation

Early local recognizers gained useful capability by narrowing what people could ask of them. Apple’s Print Recognizer shipped on Newton OS 2.0 MessagePad 120 units in December 1995. Hand printing gave it pen lifts that helped separate strokes. A neural classifier scored possible characters, and a search considered alternative stroke groupings and words, including words outside the dictionary. Microsoft’s contemporary PC speech recognizer restricted the command grammar and compressed its acoustic model. It ran in real time on a 486DX PC with 4 MB of system memory, at the cost of making users phrase commands within the grammar. Newton Print Recognizer · Microsoft’s 1995 recognizer

As mobile applications took on continuous listening and camera input, engineers had to budget storage and arithmetic separately. In 2015, Tara Sainath and Carolina Parada evaluated small keyword detectors by comparing missed detections at a fixed false-alarm rate under a multiplication budget. Google’s 2017 MobileNets work gave vision developers two practical controls over resource use, network width and input resolution. Reducing the number of stored parameters alone could leave much of the work per image intact. Small-footprint keyword spotting · MobileNets

The TensorFlow Lite developer preview of November 14, 2017 separated model development from execution on a phone. In the announcement’s architecture figure, follow the converted model file. It travels to the device, where a small interpreter executes the operations it supports. Delivery still mattered. In Kelvin Ma’s account of Google Photos, the team used a shared inference library but downloaded an editing model after installation to keep the app download small. Google renamed TensorFlow Lite to LiteRT in 2024 as the tooling expanded beyond models authored in TensorFlow. Original preview and architecture figure · Google Photos · LiteRT announcement

On a microcontroller, a model can fit in storage yet need more RAM than the device has for intermediate results. In 2020, MCUNet addressed this by designing the network and its execution engine together. The engine planned when operations could reuse a buffer, and the search used that memory plan to find networks the device could run. TensorFlow Lite Micro’s 2021 paper described a reusable interpreter for devices that might lack a filesystem or a general-purpose allocator. The application supplies an arena, a contiguous region of memory. During initialization the interpreter assigns space within it so later operations can reuse memory once earlier results are no longer needed. MCUNet · TensorFlow Lite Micro

Local language generation posed a distribution problem as well as a capacity problem. An early llama.cpp README records a four-bit, seven-billion-parameter LLaMA model generating on a MacBook CPU, with conversion tools alongside the executable. Justine Tunney combined that runtime with Cosmopolitan Libc so a user could receive the weights and executable together in one portable file. Mozilla announced the first llamafile release on November 29, 2023. A generator became easier to try on an existing laptop, while an always-listening detector still had good reason to specialize in a single phrase. Early llama.cpp snapshot · Mozilla’s release announcement · Hood and Tunney on llamafile · Mozilla

Execution and resource fit

Runtime and processor placement

A runtime can divide one model’s computation among processors on the same device. It works from a graph whose nodes are operators, such as multiplication or reshaping, and whose edges carry numerical arrays called tensors. A backend implements operators for a particular processor. A central processing unit (CPU) runs general-purpose instructions, a graphics processing unit (GPU) can perform many calculations in parallel, and a neural processing unit (NPU) specializes in neural-network arithmetic. Which nodes can run on each processor depends on the backend’s support for their operator versions, numerical types and input shapes with the installed operating system and driver. Open Neural Network Exchange (ONNX) Runtime architecture · Microsoft’s guide to observing NPU execution

ONNX Runtime asks its backends, called execution providers, which parts of the graph they can handle. For a ResNet image model, its mobile usability checker estimated that the CoreML provider could handle 120 of 122 nodes, split into two groups. The checker flags an unsupported Flatten operation and a restriction on inputs whose shape can vary. The count tells you how much of the graph is eligible. An execution trace on the device shows where the operations actually ran for a given input. Published checker output

Look at the middle operation in the partition figure. A CPU backend can execute an operation the accelerator cannot handle, but the CPU must first obtain its input. The next accelerator operation then waits for the CPU’s output. With separate memory pools, each handoff can require a copy. Shared memory can remove the copy while preserving the dependency. Even a cheap CPU operation can make the split slower than CPU-only execution. Compare both paths with matched inputs. Where no backend supports the operation, the configuration can fail outright. LiteRT GPU delegate guidance

Conceptual separate-memory example with one accelerator shown at two steps. U has a CPU implementation here. Transfers and synchronization add boundary work; the figure makes no timing comparison. Shared-memory systems may avoid explicit copies while retaining execution dependencies. Measure the complete request against the CPU-only reference.

Deployment also determines who supplies the backend. Native apps can embed it. In Emma Ning’s 2025 Foundry Local demonstration, a Microsoft service hosted models on the client computer using ONNX Runtime. Its catalog exposed hardware-specific variants. A Qualcomm NPU variant was absent on Ning’s machine, which lacked that NPU. In a browser, ONNX Runtime Web needs matching WebAssembly runtime binaries, model files and any external tensor data. Worker loading must satisfy the page’s origin and security rules, and WebGPU requires a secure context. ONNX Runtime Web deployment

Peak resident memory

A useful pair of numbers appears in Google’s FunctionGemma model card. The Mobile Actions dynamic-int8 variant occupied 288 MB as a model file and reached 551 MB peak resident set size during the device test. Resident set size (RSS) counts the process’s private and shared pages present in RAM. The test used a Samsung S25 Ultra CPU, LiteRT’s XNNPACK backend and four threads, with 512 prompt tokens, 32 generated tokens and a 1,024-token context configuration. Use the first number to plan storage. Use the second to understand process memory under those conditions. Subtracting them would not isolate a particular runtime buffer. FunctionGemma device measurements

Budget simultaneous demand in each physical pool. Include resident weights, intermediate model values called activations, runtime workspaces, app buffers and state private to each active request. Leave headroom for other applications and the operating system. Reused storage counts once while two live copies count twice. MLX on Apple silicon lets CPU and GPU operations address the same array in unified memory. For separate processor memories, charge actual copies to their respective pools. Across Android processes, proportional set size (PSS) attributes a fraction of a shared page to each process. Adding RSS values would count that page repeatedly. MLX unified memory · The capacity envelope

Request demand against usable RAM

Example

A longer input can cross the same allowance with the same weights.

Short input

3.5 GiB required, leaving 0.5 GiB free.

00.250.50.75101.252.53.755Shared pool (dimensionless)Peak demand (GiB)WeightsOther live buffersHistory cacheUsable allowanceWeights 2 GiBBuffers 1 GiBCache 0.5 GiB4 GiB allowance
  • 1. Weights
  • 2. Other live buffers
  • 3. History cache
  • 4. Usable allowance
Read coordinates and regions as data

X: 0–1 dimensionless; Y: 0–5 GiB, increasing up. Axes scaled independently; screen angles and distances are not comparable.

Weights (polygon)

(0.2, 0); (0.8, 0); (0.8, 2); (0.2, 2)

Other live buffers (polygon)

(0.2, 2); (0.8, 2); (0.8, 3); (0.2, 3)

History cache (polygon)

(0.2, 3); (0.8, 3); (0.8, 3.5); (0.2, 3.5)

Usable allowance (polyline)

(0.05, 4); (0.95, 4)

Weights 2 GiB: (0.5, 1)

Buffers 1 GiB: (0.5, 2.5)

Cache 0.5 GiB: (0.5, 3.25)

4 GiB allowance: (0.5, 4.25)

Long input

4.5 GiB required, exceeding the allowance by 0.5 GiB.

00.250.50.75101.252.53.755Shared pool (dimensionless)Peak demand (GiB)WeightsOther live buffersHistory cacheUsable allowanceWeights 2 GiBBuffers 1 GiBCache 1.5 GiB0.5 GiB over
  • 1. Weights
  • 2. Other live buffers
  • 3. History cache
  • 4. Usable allowance
Read coordinates and regions as data

X: 0–1 dimensionless; Y: 0–5 GiB, increasing up. Axes scaled independently; screen angles and distances are not comparable.

Weights (polygon)

(0.2, 0); (0.8, 0); (0.8, 2); (0.2, 2)

Other live buffers (polygon)

(0.2, 2); (0.8, 2); (0.8, 3); (0.2, 3)

History cache (polygon)

(0.2, 3); (0.8, 3); (0.8, 4.5); (0.2, 4.5)

Usable allowance (polyline)

(0.05, 4); (0.95, 4)

Weights 2 GiB: (0.5, 1)

Buffers 1 GiB: (0.5, 2.5)

Cache 1.5 GiB: (0.5, 3.55)

0.5 GiB over: (0.5, 4.75)

Illustrative simultaneous demand in one physical pool. Both requests reserve 2 GiB for resident weights and 1 GiB for other live allocations. Device, runtime, precision and concurrency are fixed. Only input length changes the growing cache. The dashed line is the allowance after system reservations.

Mapping a model file establishes a virtual address range. On Linux, accessing a mapped page can still require fetching it from storage. A mapping can therefore succeed before the application encounters expensive page faults. Request state adds a different pressure. A language model’s key-value cache retains attention data derived from earlier tokens. A growing cache uses more space as history lengthens. A static cache reserves its maximum shape up front, while a sliding-window layer can bound how much history it retains. Allocating more slots does not extend the sequence length supported by the model. Linux file mappings · The KV cache

In the illustrative memory comparison, both requests use 2 GiB of resident weights and 1 GiB of other live allocations. Only the input length changes. A full-history cache needing 0.5 GiB fits under the 4 GiB allowance. At 1.5 GiB of cache, demand reaches 4.5 GiB. The upper band shows the extra space the second request would need. For image inference, larger feature maps can increase activation storage instead. For concurrent requests, include every private cache that can be live at once before choosing how many requests to admit.

Precision and device fit

Quantization represents numerical values approximately using fewer distinct levels. On the memory plot, a weight-only change targets the bottom band. If the backend could halve its 2 GiB resident weight allocation while keeping all other allocations fixed, the longer request would need 3.5 GiB. To find the real saving, inspect the package and the execution path. Packed weights need metadata such as scales, and the backend may expand values into a wider representation for computation. Cache storage can use a separately configured precision. What quantization changes

A model can use lower precision in some layers and keep more precision where rounding would change a decision. This is mixed precision. In LinkedIn’s 360Brew ranker, the output head is the part of the model that scores candidates. The team found that lower precision there made nearly equal probabilities hard to distinguish, so they kept the head at 32-bit floating-point precision. When comparing variants for a device, run the same task cases on the same backend and inspect which decisions change. Measure peak memory, latency and energy separately. A backend that must convert packed weights before using them can save space with little gain in speed. 360Brew · Compression at the Edge · Shelby Heinecke on evaluating efficient models

Time to a usable result

Time the event the user is waiting for. A client can mark submission, request send, first response received, first useful content displayed and completion. For a text model, prefill processes the prompt and decode produces successive output tokens. A long prompt can delay response onset even when decoding is fast. In streaming measurements, one response chunk can contain several tokens. A user waiting for a proposed command may need the whole command before any of it is useful. Prefill and decode · Latency has boundaries

Watch the reply grow in Adrien Grondin’s offline iPhone demonstration. He reports about 40 generated tokens per second for his four-bit Gemma 4 configuration with MLX. That rate describes output once generation is underway. A timestamp at submission and another at first usable display would be needed to measure the initial wait. A prolonged run would answer a third question about sustained speed.

What occupies the submit-to-display interval

Example timings

Halving accelerator execution saves 15 ms of the 120 ms wait.

Submit to usable display0–120 msDuration 120 ms
Wait for admission0–20 msDuration 20 msWithin Submit to usable display
Prepare input20–50 msDuration 30 msWithin Submit to usable display
Copy input to accelerator50–65 msDuration 15 msWithin Submit to usable display
Compute model output65–95 msDuration 30 msWithin Submit to usable display
Return output95–105 msDuration 10 msWithin Submit to usable display
Interpret and display105–120 msDuration 15 msWithin Submit to usable display
For this single request with separate memory pools, child stages run sequentially. The queue and transfers delay display even while the model is idle. Read elapsed time from the parent span. Adding overlapping intervals, including the parent and its children, double-counts time.
Read the diagram as text
  • Submit to usable display. 0 to 120 ms; duration 120 ms.
  • Wait for admission. 0 to 20 ms; duration 20 ms. Parent: Submit to usable display.
  • Prepare input. 20 to 50 ms; duration 30 ms. Parent: Submit to usable display.
  • Copy input to accelerator. 50 to 65 ms; duration 15 ms. Parent: Submit to usable display.
  • Compute model output. 65 to 95 ms; duration 30 ms. Parent: Submit to usable display.
  • Return output. 95 to 105 ms; duration 10 ms. Parent: Submit to usable display.
  • Interpret and display. 105 to 120 ms; duration 15 ms. Parent: Submit to usable display.

Trace queueing and preparation as well as processor time. In the example timeline, accelerating a 30 ms model computation by a factor of two removes 15 ms from a 120 ms wait. The copies and display work still take time. Even inside the model, peak arithmetic throughput is relevant only when arithmetic limits progress. If the processor spends most of its time fetching weights for one user’s next token, memory bandwidth, measured in bytes moved per second, can limit the output pace. Roofline performance model

For continuous media, compare service rate with the arriving stream. Speech recognition’s real-time factor is processing time divided by audio duration. Eight seconds to process ten seconds of audio gives 0.8. A warm test on complete audio buffers can achieve that rate while live dictation still displays words late or revises them for too long. Measure partial and final text separately. For video, count frames processed before their deadline and frames skipped. Bound input size, queue length and concurrency according to those deadlines. A frame processed after it can influence the next decision consumes resources without serving its purpose. Speech measurement boundary · Scheduling and admission

Delivery and lifetime

Installing a compatible model package

Package the capability that passed the test. A manifest identifies one release’s weights, architecture, input processor, output conventions, compatible runtime and distribution terms. A chat model can load with the wrong vocabulary or message template and still produce text. The vocabulary assigns token identifiers, while the template inserts the boundaries that tell the model who spoke. Changing either can change the meaning of its input. An image classifier likewise depends on expected pixel normalization and a matching list of output labels. Run a known input through preparation, execution and output interpretation to catch a mixed package. Tokenization · LiteRT model metadata

Bundling delivers assets at app installation. An on-demand download can supply only the features or languages the user needs, but arriving bytes may still need preparation. In Apple’s Core ML workflow, the downloaded .mlmodel is compiled into .mlmodelc before the app creates a usable model. The compiled output starts in temporary storage. Moving it to persistent storage allows reuse at the next launch. Allow disk space for the currently active release, the incoming release and compilation output during the switch. Apple’s download and compilation guide

Activating a complete package

Example

A validated replacement can exist on disk before any new session uses it.

1 / 3 · Stage the replacement

Transfer B outside A’s directory. An interruption leaves A selected.

New this stepRelease AWeights, input processor and configuration in directory A.
New this stepRelease BSeparate destination for the new compatible set.
New this stepActive record selects A
New this stepStaged transfer
In this installer design, A serves throughout staging and verification of B. One atomic change to the selection record lets new sessions open B. Existing sessions retain access to the complete release they opened.
Read the diagram as text
  • Release A. Weights, input processor and configuration in directory A.
  • Release B. Separate destination for the new compatible set.
  • Active record selects A.
  • Staged transfer.
  • B passed validation. Complete, authenticated and tested together.
  • Active record selects B.
  • Retained for existing readers.
  • Active record selects A → Release A: New sessions open.
  • Staged transfer → Release B: Writes replacement files.
  • B passed validation → Release B: Checks apply to this set.
  • Active record selects B → Release B: New sessions open.
  • Retained for existing readers → Release A: Keeps old files accessible.
  1. Stage the replacement. Transfer B outside A’s directory. An interruption leaves A selected. Active: Release A, Release B, Active record selects A, Staged transfer. New: Release A, Release B, Active record selects A, Staged transfer.
  2. Verify B as a set. New sessions still open A while B becomes eligible for use. Active: Release A, Release B, Active record selects A, B passed validation. New: B passed validation.
  3. Commit the active record. Select B for new sessions. Readers that opened A can finish with A. Active: Release A, Release B, B passed validation, Active record selects B, Retained for existing readers. New: Active record selects B, Retained for existing readers.

Before activating a download, verify who authorized its contents. The Update Framework starts from trusted keys, authenticates release metadata and checks target file lengths and hashes. Metadata versions and expiry checks also help detect rollback, inconsistent sets and stale metadata. Then verify compatibility and behavior on the device. A correctly signed package can still require a runtime feature the installed version lacks. The Update Framework specification · AI Security

The installation figure shows a design for surviving an interrupted update. Each release has its own directory. A session reads one active-version record and resolves all its files from that release. In the middle step B is complete and tested, yet new sessions still open A. Replacing the small record atomically selects B. Android’s AtomicFile can commit that one file. The installer must coordinate writers, finish and persist the referenced files first, and keep A while existing readers need it. A later recovery can select A only while that complete release remains compatible and authorized. AtomicFile contract

Cold starts, readiness and residency

In the Foundry Local demonstration, Ning used foundry cache list to inspect models she had downloaded before presenting. This removed the need to acquire those models during the demonstration. A disk inventory answers a different question from whether the next call can run immediately. Loading weights, creating the execution session, compiling for a target and warming up can still precede a first result. Eager loading moves that delay before demand. Lazy loading avoids the work entirely when the feature goes unused. Keeping a session resident saves initialization on subsequent calls and occupies memory between them.

Measure readiness from the entry condition the user actually encounters.
Entry conditionWhat survivesWhat the next request needs
First installationApp and bundled assetsAcquire missing assets, verify and prepare the release, then initialize.
Installed cold startComplete release and valid persistent compilation outputLoad or rebuild the session locally and warm it as required.
Warm requestCompatible resident sessionPrepare the new input and execute.
Mandatory asset purgedAn incomplete setRestore the missing file and validate readiness before serving.

Recovery depends on which resource disappeared. Android can terminate an application under memory pressure. Chrome can freeze a tab or discard it without a final cleanup callback. Save recoverable task inputs and progress before either event. Reopening the client can reconstruct a session from surviving assets even though the interrupted computation is gone. Test session creation again after accelerator loss. Chrome’s managed models can also be purged under disk pressure or changed eligibility, in which case the app has to initiate another download. Chrome page lifecycle · Built-in model management

Cancellation also has a scope. Chrome’s Prompt API accepts an abort signal for one prompt. Destroying the session aborts its work and releases session resources, while the lifetime of shared weights is a separate matter. In ONNX Runtime’s Java API, setTerminate(true) targets every incomplete call using the same RunOptions instance. Give independent requests separate cancellation scopes and test that cancelling one allows the other to finish. Chrome session management · ONNX RunOptions · Design Engineering and AI Interfaces

Sustained operation

Energy, heat and sustained work

Power is the rate of energy use. For example, a task averaging 2 watts over 3 seconds uses 6 joules. Finishing the same task in 1 second at 3 watts uses 3 joules. To compare battery demands, measure energy over the intended period of use and divide by useful completions. Include sensing, memory traffic and idle intervals that the feature requires. A thermal budget is the heat load the device can sustain under specified conditions. When heat accumulates, the device can reduce performance to control its temperature. This throttling changes both the time per task and the amount of work completed.

Apple’s 2017 iPhone wake detector illustrates a way to change the duty cycle, the fraction of time a component works. A low-power auxiliary processor checked audio continuously. Only a strong candidate woke the main processor for a more expensive check. Follow the weak-detection branch in the figure. The phone keeps listening while avoiding the larger computation for that candidate. Tightening the first threshold can save work but can also discard a genuine activation before the second detector sees it. Apple’s Hey Siri engineering account

Which detections wake the main processor

Only candidates that pass the low-power check incur the larger local check.

Microphone
Auxiliary detector
End candidate check
Main processor checks candidate
Reject activation
Accepted local activation
Siri server
In Apple’s 2017 iPhone design, the auxiliary detector listened continuously. A strong candidate woke the main processor. Audio accepted by the local checks could then reach Siri’s server for further processing.
Read the diagram as text
  • Microphone.
  • Auxiliary detector.
  • End candidate check.
  • Main processor checks candidate.
  • Reject activation.
  • Accepted local activation.
  • Siri server.
  • Microphone → Auxiliary detector: Audio samples.
  • Auxiliary detector → End candidate check: Weak detection.
  • Auxiliary detector → Main processor checks candidate: Strong detection triggers wake.
  • Main processor checks candidate → Reject activation: Second check fails.
  • Main processor checks candidate → Accepted local activation: Second check passes.
  • Accepted local activation → Siri server: Accepted audio if sent.

Repeated requests can reveal a slowdown that a single request misses. The MELT researchers ran four-bit Zephyr-3B through 50 consecutive prompts on an iPhone 14 Pro and repeated the experiment three times. They observed declining throughput and proposed changes in the phone’s power management and clock frequency as a possible cause. In a chat app, a growing conversation can also slow generation because each new token can require reading more history. To distinguish these effects, save a short history and a long one. Replay each on the same phone both cool and warm, keeping the model loaded and the backend, output length and concurrency fixed. Compare temperatures using the same history, then compare histories at the same temperature. MELT study

Record ambient temperature, charging and battery state, competing applications, thermal status and timely completions over the same observation window. State which power rails were measured. Android’s Power Profiler observes device activity, so another app can contribute to those readings. If the workload is unsustainable, choose the concession the task allows. Sampling less often can miss a brief event. An output limit may leave a response incomplete. Deferral can work for a batch with a later deadline, provided the runtime is available then. ML Kit GenAI requires the top foreground app. Plugging in the phone while another app occupies the screen does not make that managed API available to a background job. Android Power Profiler · ML Kit GenAI resource contract

Connectivity and cooperation

What works offline

Offline inference needs no live network connection to execute the model. An offline feature must also acquire its input, interpret the result and complete its promised action with locally available dependencies. Test a fresh installation separately from a provisioned cold start. For the latter, stop the model-hosting process as well as the client, disconnect, then request the feature. Transformers can load a previously downloaded model with local_files_only=True. A browser deployment may also need a cached runtime binary and external tensor files. Neither loader supplies an unavailable tool or refreshes an expired entitlement. Transformers offline loading

In Brick’s LiteRT-LM workshop, one skill could run local JavaScript, while a music skill required a key and a web service. A prototype Wikipedia archiver fetched articles and indexed them ahead of time. After disconnection it could search a saved article but could neither supply a missing article nor discover a later edit. Give users the collection date and scope when the available result depends on them. The same limit applies to authority. A disconnected app can enforce a local expiry, but it cannot immediately learn that a remote party revoked access. Search and Retrieval · Enforce current authority

Disconnect before startup, during use and after a send. The outcome depends on what the application had already obtained or transmitted.
ConditionOutcome the user can rely onBehavior on reconnection
Complete installation, valid local authority and local actionComplete the task and save its result locally.No transfer is needed.
Missing mandatory asset or expired required authorityIdentify the unavailable capability. Preserve any permitted draft.Acquire the asset or re-establish authority before proceeding.
Older reference collectionOffer a dated result if the task permits stale data.Refresh according to the collection’s policy.
Remote action requested while offlineOffer a draft, or a pending send if the later transfer is explicitly authorized.Check that authorization still applies before sending.
Connection lost after transmitting a writeMark the effect uncertain and retain its operation identifier.Reconcile with the recipient before retrying. The original attempt may have succeeded.

Authorized cloud handoffs

A workflow can deliver a useful local result before offering a remote operation. Google’s 2019 Recorder implementation transcribed speech, indexed words and retained their timestamps on the device. Searching for a word could jump to the corresponding audio without internet access. Today’s Recorder help treats summarizing a saved transcript and re-transcribing audio as separate operations with their own eligibility and cloud-processing conditions. Permission and connectivity requirements therefore attach to the particular operation. Recorder engineering account · Summary and re-transcription guidance

Consider a proposed notebook that offers to summarize only a selected passage. It can show the passage, identify the summary service and explain that service’s use and retention of the text. Check permission for that transfer before invoking the service. In the diagram, the transcript is already usable on every branch. Denial keeps the selection local. With approval but no connection, the notebook can report that nothing was sent. When approved and connected, it sends the selected text, receives the summary and identifies its remote origin. Microphone permission covers capture. The outbound passage needs its own authorization. Media Capture and Streams · Approve actual service paths

A remote summary of selected text

Example

Only the selected passage has a permitted route to the summary service.

Local audio
Local timestamped transcriptAn independently useful result.
Check proposed sendMatch selected text, recipient, purpose and retention to approval, then check the connection.
Keep selection private
Report passage unsentNo automatic upload is scheduled.
Approved remote summary serviceOnly the recipient identified in the permission check.
Summary displayed on device
In this proposed notebook, local speech recognition produces a transcript the user can keep on every branch. Without approval or connectivity the passage stays unsent. With both, the identified service receives the approved text and returns a summary for display.
Read the diagram as text
  • Local audio.
  • Local timestamped transcript. An independently useful result.
  • Check proposed send. Match selected text, recipient, purpose and retention to approval, then check the connection.
  • Keep selection private.
  • Report passage unsent. No automatic upload is scheduled.
  • Approved remote summary service. Only the recipient identified in the permission check.
  • Summary displayed on device.
  • Local audio → Local timestamped transcript: Local recognition yields words and times.
  • Local timestamped transcript → Check proposed send: Selected passage.
  • Check proposed send → Keep selection private: Stop if approval absent.
  • Check proposed send → Report passage unsent: Stop if approved but offline.
  • Check proposed send → Approved remote summary service: Selected text if approved and reachable.
  • Approved remote summary service → Summary displayed on device: Returned summary text.

Changing the representation can reduce what is sent while preserving sensitive information. Selected words can name a person. A numerical embedding can preserve clues to the original text. Researchers with target embeddings, a trained inversion model and query access to the embedding model recovered names from embedded clinical notes. A different placement technique, split inference, divides the numerical operations of one model between machines. The 2017 Neurosurgeon study explored sending intermediate arrays at such a split because network conditions could change whether offloading saved time or device energy. Each design needs an inventory of what its recipient actually receives. Embedding inversion study · Embeddings and Representation Learning · Distributed Training and Inference

A third design routes the whole request according to local capability. In Florina Muntenescu and Oli Gaymond’s Gemini Nano account, cloud execution expands coverage to devices without the local model. Firebase AI Logic distinguishes unavailable, downloadable, downloading and available states. Submitting a request does not download the model. ONLY_ON_DEVICE fails when local execution cannot proceed. PREFER_ON_DEVICE allows fallback, including for unsupported features. Check the permitted recipient before invoking that mode and show the execution location reported in the response. Resource pressure can make a route infeasible, but only the application’s data policy can authorize another recipient. Google DeepMind · Firebase hybrid inference · Model Routing and LLM Gateways

Deployment qualification

Evidence for device support

Qualify a versioned capability under named conditions. For each device and OS group, record the package revision, input processor, runtime, actual backend, admitted input and output sizes, concurrency and entry state. Include the lowest-memory target. Set task criteria and deadlines first, use development cases to choose a configuration, then assess held-out cases. A faster run that omitted a required part of the answer failed the task. Evals and Benchmarks · Benchmark the workload

Use a separate set of rows for each device, OS, package and backend combination. Leave an untested condition visible as untested.
PromiseExercisePassing observationDecision if unmet
Task usefulnessHeld-out inputs across supported languages and sizesOutputs meet the error and completeness criteria.Narrow the task or change the model.
Execution targetActual device, drivers and input shapesOperator trace confirms placement and results pass.Qualify another backend separately.
Installation and recoveryInterrupt an update, kill the host, restart offline, remove a required fileA complete supported release runs when promised. Missing dependencies are identified.Repair provisioning or restrict the offline promise.
Memory fitLargest inputs and maximum admitted overlapPeak in every pool fits its usable allowance.Limit request size, residency or concurrency.
ResponsivenessCold and warm requests at the intended arrival rateFirst useful display and completion meet their respective deadlines.Trace the slow stage or restrict the mode.
Sustained workRepeated tasks under stated power, thermal and background conditionsTimely useful completions stay within the energy allowance.Reduce duty cycle or defer eligible work.
Data boundariesDeny a transfer, disconnect mid-request, reconnect and exercise backupObserved recipients, fields and deferred effects match the policy.Disable the unqualified path.

Keep the measurement window with the result. MLPerf Tiny’s 2021 procedure paired embedded tasks with quality targets and reported energy per inference in a window that excluded preprocessing and postprocessing. MobileAIBench evaluated task quality separately from its iPhone measurements of resource use and latency. A fast recognizer inside either kind of test would still need a measurement from microphone input to finalized text to qualify live dictation. MLPerf Tiny · MobileAIBench

Network evidence also has a boundary. A browser service worker can answer a page request from cache or make a fetch of its own. Disabling the worker changes the application being tested. Page observations alone cannot account for a native service or an OS backup. Record the route, version, stage and duration needed to diagnose failures while leaving user content out of routine diagnostics. A defensible release decision for the notebook would be to enable foreground transcription on configurations that passed, preserve the transcript when summarization is unavailable, and offer transfer only on a separately authorized and tested summary path. Service-worker observations · Observability

Frequently asked questions

Can an API call run entirely on my laptop?

Yes. Select a downloaded model on the laptop’s Ollama server.

Can a downloaded model start offline?

Yes, if the cold session can initialize using local files. Test with the hosting process stopped and the network disconnected. Include the runtime, tokenizer or media processor, and any external tensor files or browser runtime binaries the chosen package needs.

How soon will a streaming answer be useful?

Measure from submission to the first displayed content that satisfies the immediate need. A token rate measures output pace, while a first-response timestamp measures arrival at the client. Both can miss a wait for the app to assemble or display a usable result.

What happens when the local model is unavailable?

A local-only feature can report the missing capability or offer a qualified local alternative. A local-preferred feature can use a reachable cloud service if the payload, purpose and recipient are authorized. Check that permission before a fallback-capable request and report where it ran.

Open questions

  1. How accurately can short tests predict useful work through an hour of mobile use? History length, competing applications and heat can all change output rate. Stock-device studies aligning timely completions, memory, energy and thermal state under controlled conditions would help teams set sustainable input and duty-cycle limits.

  2. How long can a user rely on a model prepared before going offline? The platform can remove a session or purge an asset, and the two losses have very different recovery requirements. A verifiable retention interval or timely notice of impending loss would make advance provisioning more dependable.

  3. Which representation discloses the least information while preserving a remote stage’s usefulness? Shortening a passage or sending a vector changes what the service can compute and what it may reconstruct. Matched task tests and reconstruction tests under the actual recipient’s access assumptions would give teams a basis for choosing a handoff.

Follow the curated reading path through the speakers and demonstrations behind this entry.

44 min

AI Engineer World's Fair 2026 · 2026

State of the Union: Why Local, Why Now

Nader Khalil · Alex Cheema · Matthew Berman · Ahmad Osman · Joseph Nelson

Cited in this entry

Joseph Nelson’s robot vision example shows how unreliable connectivity changes the capability worth putting on a machine.

Watch talk
18 min

AI Engineer World's Fair 2024 · 2024

A Practical Guide to Efficient AI

Shelby Heinecke

Cited in this entry

Shelby Heinecke connects a smaller weight representation to the tests still needed for shipping it, including task quality, safety and device resource use.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

128 matching talks

TalkSpeakerEventYear
Florina Muntenescu, Oli GaymondAI Engineer Europe 20262026
Adrien GrondinAI Engineer Europe 20262026
Mozhgan Kabiri ChimehAI Engineer Europe 20262026
Compilers in the Age of LLMs

Transcript reviewed

Yusuf OlokobaAI Engineer Code 20252025
Cormac BrickAI Engineer World's Fair 20262026
Ishan AnandAI Engineer World's Fair 20252025
Philip Kiely, Pankaj GuptaAI Engineer World's Fair 20242024
Phil NashAI Engineer Europe 20262026
Mark MoyouAI Engineer World's Fair 20242024
Dylan PatelAI Engineer World's Fair 20242024
Stephen Hood, Justine TunneyAI Engineer World's Fair 20242024
Mithun HunsurAI Engineer Summit 20232023
Sarah ChiengAI Engineer Europe 20262026
Ziv IlanAI Engineer Europe 20262026
Hamed Firooz, Maziar SanjabiAI Engineer World's Fair 20252025
Vibhu SapraAI Engineer World's Fair 20252025
Jerry LiuAI Engineer Summit 20232023
Roy DerksAI Engineer Summit 20252025
Abi AryanAI Engineer Summit 20232023
Simon WillisonAI Engineer World's Fair 20242024
Fouad MatinAI Engineer World's Fair 20252025
How to Build Trustworthy AI

Transcript reviewed

Allie HoweAI Engineer World's Fair 20252025
Lukas BiewaldAI Engineer World's Fair 20242024
Ibragim BadertdinovAI Engineer Europe 20262026
Sayash KapoorAI Engineer Summit 20252025
Walden, Carter, Tanay, Alex Atallah, NavAI Engineer World's Fair 20262026
Lech KalinowskiAI Engineer World's Fair 20262026
Kyle KranenAI Engineer World's Fair 20252025
Rishabh BhargavaAI Engineer Europe 20262026
Building a Chess Coach

Transcript reviewed

Anant Dole, Asbjørn SteinskogAI Engineer Europe 20262026
Steve KorshakovAI Engineer World's Fair 20262026
Tun Shwe, Jeremy FrenayAI Engineer Europe 20262026
Ben BurtenshawAI Engineer Europe 20262026
Dmytro (Dima) DzhulgakovAI Engineer World's Fair 20242024
AI Engineer Summit 20252025
Gabriel Jorge MenezesAI Engineer World's Fair 20262026
Keegan McCallumAI Engineer World's Fair 20252025
Vinoth GovindarajanAI Engineer World's Fair 20262026
Alex AtallahAI Engineer World's Fair 20252025
Mike ChambersAI Engineer World's Fair 20252025
Audry HsuAI Engineer Europe 20262026
Sandipan BhaumikAI Engineer Europe 20262026
Bilge YücelAI Engineer Europe 20262026
Rachel Lee Nabors (RL Nabors)AI Engineer World's Fair 20262026
Charles FryeAI Engineer World's Fair 20252025
Why MLX

Transcript reviewed

AI Engineer Europe 20262026
Matthias LoiblAI Engineer World's Fair 20252025
Chintan Parikh, Weiyi WangAI Engineer Europe 20262026
Nick TaylorAI Engineer Europe 20262026
Maxime LabonneAI Engineer Europe 20262026
Stefano FiorucciAI Engineer Europe 20262026
Brendan O'DonoghueAI Engineer Europe 20262026
Sean DuBois, Kwindla Hultman Kramer, YaxinAI Engineer World's Fair 20252025
Chintan Agrawal, Daniel WirjoAI Engineer World's Fair 20262026
Tushar JainAI Engineer World's Fair 20262026
Notion's Token Town

Transcript reviewed

Sarah SachsAI Engineer World's Fair 20262026
Charles FryeAI Engineer World's Fair 20252025
Kelvin MaAI Engineer World's Fair 20252025
Alex CheemaAI Engineer Europe 20262026
Compression at the Edge

Cited in this entry

Chris Alexiuk, Daniel Han, Asma Beevi, Merve Noyan, Parth SareenAI Engineer World's Fair 20262026
Neil Dwyer, Jack DwyerAI Engineer World's Fair 20252025
Sally Ann O'MalleyAI Engineer Europe 20262026
Shafik Quoraishee, Joanne SongAI Engineer World's Fair 20262026
Carter Abdallah, Vincent Weisser, Lucas Atkins, Chris AlexiukAI Engineer World's Fair 20262026
Angelos PerivolaropoulosAI Engineer Europe 20262026
Rajkumar SakthivelAI Engineer World's Fair 20262026
Joseph NelsonAI Engineer Summit 20232023
Ievgen VakulenkoAI Engineer World's Fair 20242024
Ezra Tanzer, Dan ArpinoAI Engineer World's Fair 20262026
Leonie MonigattiAI Engineer Europe 20262026
Armanas PovilionisAI Engineer World's Fair 20262026
Charles FryeAI Engineer Summit 20232023
Philipp SchmidAI Engineer World's Fair 20252025
AI Platform Engineering

Metadata candidate

Patrick DeboisAI Engineer World's Fair 20242024
Nagkumar Arkalgud, Keiji KanazawaAI Engineer World's Fair 20252025
Niklas NielsenAI Engineer Summit 20232023
Paul Klein IVAI Engineer World's Fair 20262026
Paige Bailey, Guillaume Vernade, Ian BallantyneAI Engineer Europe 20262026
Jeff NgAI Engineer World's Fair 20262026
Du’An Lightfoot, Banjo ObayomiAI Engineer World's Fair 20252025
Mahesh MuragAI Engineer Summit 20252025
Thor Schaeff, Philipp SchmidAI Engineer Europe 20262026
Eric ZakariassonAI Engineer Europe 20262026
Abed MatiniAI Engineer World's Fair 20262026
Jedrick Kosinski, ComfyAnonymousAI Engineer World's Fair 20252025
Liam HamptonAI Engineer Europe 20262026
Hanchi WangAI Engineer World's Fair 20242024
Defying Gravity

Metadata candidate

Kevin HouAI Engineer Code 20252025
Ben HylakAI Engineer World's Fair 20262026
Gaurav MishraAI Engineer World's Fair 20262026
Antje Barth, Mike ChambersAI Engineer World's Fair 20242024
KitzeAI Engineer World's Fair 20252025
Frontier Feud

Metadata candidate

Barr Yaron, Mihir, John, Tina, Shresta, Paige, Colin, Petra, StevenAI Engineer Summit 20252025
Cassidy HardinAI Engineer Europe 20262026
Dave Burnison, Alex Malebranche, Dimitrios Philliou, Christina Warren, HaraldAI Engineer World's Fair 20242024
John PhamAI Engineer World's Fair 20252025
Kyle Jaejun LeeAI Engineer World's Fair 20262026
Ritvik PandyaAI Engineer World's Fair 20262026
Juan PeredoAI Engineer Summit 20252025
2025 in LLMs so far

Metadata candidate

Simon WillisonAI Engineer World's Fair 20252025
Ronan McGovernAI Engineer World's Fair 20252025
Stefania DrugaAI Engineer World's Fair 20262026
Vikhyat KorrapatiAI Engineer World's Fair 20242024
Neil ZeghidourAI Engineer Europe 20262026
Simon WillisonAI Engineer Summit 20232023
Juan Herreros ElorzaAI Engineer Europe 20262026
Hursh AgrawalAI Engineer World's Fair 20262026
Jon PeckAI Engineer World's Fair 20252025
Pamela Fox, Harald Kirschner, Gabriela de QueirozAI Engineer World's Fair 20242024
Merve NoyanAI Engineer Europe 20262026
Arjun Desai, Rohit TalluriAI Engineer World's Fair 20252025
Gus Martins, Ian BallantyneAI Engineer Europe 20262026
Thiyagarajan MaruthavananAI Engineer World's Fair 20262026
Cedric ClyburnAI Engineer World's Fair 20262026
Ahmad OsmanAI Engineer World's Fair 20262026
The Log Is The Agent

Metadata candidate

Ishaan SehgalAI Engineer World's Fair 20262026
Arturo NunezAI Engineer World's Fair 20262026
Jonathan FernandesAI Engineer World's Fair 20252025
Thabang LedwabaAI Engineer World's Fair 20252025
Kathleen KenealyAI Engineer World's Fair 20242024
Nico AlbaneseAI Engineer Summit 20252025
Harald KirschnerAI Engineer World's Fair 20252025
Lucas PalmaAI Engineer World's Fair 20262026
Fryderyk Wiatrowski, Peter AlbertAI Engineer World's Fair 20242024
Subbiah Sethuraman, Abhilash AsokanAI Engineer World's Fair 20262026
Du’An Lightfoot, Khaja OmerAI Engineer World's Fair 20262026
Karthic RaoAI Engineer NYC 2026 Online Talks2026
Paige BaileyAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
67 processed in full · 6 in the curated path
Automated source review
Passed
Metadata candidates
67 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. vLLM GPU worker: profiling available KV capacity

    https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/worker/gpu_worker.py

  2. Transformers: cache allocation versus retained history

    https://huggingface.co/docs/transformers/en/kv_cache

  3. MLX: Unified memory

    https://ml-explore.github.io/mlx/build/html/usage/unified_memory.html

  4. Ollama API introduction

    https://docs.ollama.com/api/introduction

  5. Ollama API introduction, API example

    https://docs.ollama.com/api/introduction#ollama-api-example

  6. Ship it! Building Production-Ready Agents

    https://www.youtube.com/watch?v=HT4l0DeP69I&t=150

  7. llama.cpp project and quick start

    https://github.com/ggml-org/llama.cpp

  8. NIST SP 500-325: Fog Computing Conceptual Model

    https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.500-325.pdf

  9. From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google

    https://www.youtube.com/watch?v=-TiET_K-E_g&t=119

  10. OpenClaw in Your Hand: Building a Physical AI Terminal for Local LLM Agents

    https://www.youtube.com/watch?v=akk6KRlcwW4&t=140

  11. From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google

    https://www.youtube.com/watch?v=-TiET_K-E_g&t=178s

  12. State of the Union: Why Local, Why Now

    https://www.youtube.com/watch?v=KB41dTlX1Uc&t=677s

  13. What does it take to build a personal, local, private AI Agent that augments you deeply?

    https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=383s

  14. Frontier results, on device - RL Nabors, Arize

    https://www.youtube.com/watch?v=fWXJM-J0ZB8&t=410

  15. FinOps terminology: ownership, depreciation and utilization

    https://framework.finops.org/assets/terminology/

  16. FinOps: comparable workload denominators and cost boundaries

    https://www.finops.org/framework/capabilities/unit-economics/

  17. Android: Back up user data with Auto Backup

    https://developer.android.com/identity/data/autobackup?hl=en

  18. Pixel Phone Help: Save & share recordings & transcripts

    https://support.google.com/pixelphone/answer/16267696?hl=EN

  19. Hugging Face Hub: Environment variables

    https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables

  20. Ollama: FAQ

    https://docs.ollama.com/faq

  21. Linux Yama: process-memory inspection and privileged tracing

    https://docs.kernel.org/admin-guide/LSM/Yama.html

  22. Convolutional Neural Networks for Small-footprint Keyword Spotting

    https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43969.pdf

  23. Hey Siri: An On-device DNN-powered Voice Trigger for Apple’s Personal Assistant

    https://machinelearning.apple.com/research/hey-siri

  24. Scikit-learn: scoring rules and baseline estimators

    https://scikit-learn.org/stable/modules/model_evaluation.html

  25. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

    https://arxiv.org/html/2402.14905v1

  26. MCP tools: application security responsibilities

    https://modelcontextprotocol.io/specification/2025-06-18/server/tools

  27. Hugging Face model cards: locating license terms

    https://huggingface.co/docs/hub/model-cards

  28. Llama 3.1 Community License: concrete review dimensions

    https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/LICENSE

  29. llama.cpp: LLM inference in C/C++

    https://github.com/ggml-org/llama.cpp

  30. Combining Neural Networks and Context-Driven Search for On-Line, Printed Handwriting Recognition in the Newton

    https://shinyverse.org/larryy/Yaegeretal.AIMag.pdf

  31. Microsoft Windows Highly Intelligent Speech Recognizer: Whisper

    https://www.microsoft.com/en-us/research/wp-content/uploads/1995/01/1995-huang-icassp.pdf

  32. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

    https://arxiv.org/html/1704.04861v1

  33. Announcing TensorFlow Lite

    https://developers.googleblog.com/en/announcing-tensorflow-lite/

  34. Magic Editor Under the Hood: Weaving Generative AI into a Billion-User App

    https://www.youtube.com/watch?v=C13jiFWNuo8&t=193

  35. TensorFlow Lite is now LiteRT

    https://developers.googleblog.com/en/tensorflow-lite-is-now-litert/

  36. MCUNet: Tiny Deep Learning on IoT Devices

    https://papers.nips.cc/paper_files/paper/2020/file/86c51678350f656dcc7f490a43946ee5-Paper.pdf

  37. TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems

    https://proceedings.mlsys.org/paper_files/paper/2021/file/6c44dc73014d66ba49b28d483a8f8b0d-Paper.pdf

  38. llama.cpp early repository snapshot at 920a7fe

    https://github.com/ggml-org/llama.cpp/tree/920a7fe2d94cc7e4fed0e88db830b674c91865c5

  39. Introducing llamafile

    https://hacks.mozilla.org/2023/11/introducing-llamafile/

  40. ONNX Runtime Architecture

    https://onnxruntime.ai/docs/reference/high-level-design.html

  41. Develop AI applications for Copilot+ PCs

    https://learn.microsoft.com/en-us/windows/ai/npu-devices/

  42. CUDA Programming Guide: Programming Model

    https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/programming-model.html

  43. ONNX Runtime: Model Usability Checker

    https://onnxruntime.ai/docs/tutorials/mobile/helpers/model-usability-checker.html

  44. GPU delegates for LiteRT

    https://developers.google.com/edge/litert/performance/gpu

  45. Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime and Olive — Emma Ning, Microsoft — Emma Ning, Tsavo Knott

    https://www.youtube.com/watch?v=l614N5W60ls&t=282s

  46. Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime and Olive — Emma Ning, Microsoft

    https://www.youtube.com/watch?v=l614N5W60ls&t=282

  47. Deploying ONNX Runtime Web

    https://onnxruntime.ai/docs/tutorials/web/deploy.html

  48. FunctionGemma model card

    https://ai.google.dev/gemma/docs/functiongemma/model_card

  49. Android: Memory allocation among processes

    https://developer.android.com/topic/performance/memory-management

  50. Linux mmap(2) manual

    https://www.man7.org/linux/man-pages/man2/mmap.2.html

  51. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=4576

  52. Quantize ONNX models — ONNX Runtime

    https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html

  53. ONNX QuantizeLinear: quantization granularity

    https://onnx.ai/onnx/operators/onnx__QuantizeLinear.html

  54. From model weights to API endpoint with TensorRT-LLM

    https://www.youtube.com/watch?v=Lko9lTGD_9U&t=3411

  55. 360Brew: LLM-based Personalized Ranking and Recommendation — Hamed Firooz and Maziar Sanjabi, LinkedIn AI

    https://www.youtube.com/watch?v=U0S6CfzAY5c&t=811

  56. Compression at the Edge

    https://www.youtube.com/watch?v=J4_jCrTxMkk&t=707s

  57. A Practical Guide to Efficient AI

    https://www.youtube.com/watch?v=Yyg_BoeB2LU&t=978

  58. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

    https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=378

  59. GenAI-Perf: latency and throughput measurement

    https://github.com/triton-inference-server/perf_analyzer/tree/main/genai-perf

  60. Running Gemma 4 On-Device: 40 Tokens/s on iPhone with MLX — Adrien Grondin

    https://www.youtube.com/watch?v=a2muGkT4WD4&t=370s

  61. Roofline Performance Model — NERSC

    https://docs.nersc.gov/tools/performance/roofline/

  62. Frontier AI at Home (literally)

    https://www.youtube.com/watch?v=ESbWpPT_9-o&t=675

  63. RIVA ASR Developer Pack Overview

    https://docs.nvidia.com/ace-for-games/Riva-asr/2.0/developer-pack.html

  64. ML Kit GenAI Speech Recognition API

    https://developers.google.com/ml-kit/genai/speech-recognition/android

  65. Google SRE: Handling Overload

    https://sre.google/sre-book/handling-overload/

  66. Transformers: model-specific chat serialization

    https://huggingface.co/docs/transformers/en/chat_templating

  67. Adding metadata to LiteRT models

    https://developers.google.cn/edge/litert/conversion/tensorflow/metadata

  68. Core ML: Downloading and Compiling a Model on the User’s Device

    https://developer.apple.com/jp/documentation/coreml/core_ml_api/downloading_and_compiling_a_model_on_the_user_s_device/

  69. Understand built-in model management in Chrome

    https://developer.chrome.com/docs/ai/understand-built-in-model-management

  70. The Update Framework: authenticated update workflow

    https://theupdateframework.github.io/specification/latest/

  71. Android AtomicFile

    https://developer.android.com/reference/android/util/AtomicFile

  72. Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime and Olive — Emma Ning, Microsoft

    https://www.youtube.com/watch?v=l614N5W60ls&t=596

  73. LiteRT: Benchmark Interpreter API

    https://developers.google.com/edge/litert/models/measurement

  74. Chrome Prompt API: stopping prompts and managing sessions

    https://developer.chrome.com/docs/ai/prompt-api

  75. Stable Diffusion with Core ML on Apple Silicon

    https://github.com/apple/ml-stable-diffusion

  76. Android: Processes and app lifecycle

    https://developer.android.com/guide/components/activities/process-lifecycle

  77. Chrome: Page Lifecycle API

    https://developer.chrome.com/docs/web-platform/page-lifecycle-api

  78. ONNX Runtime Java API: OrtSession.RunOptions

    https://onnxruntime.ai/docs/api/java/ai/onnxruntime/OrtSession.RunOptions.html

  79. Dissecting the Impact of Mobile DVFS Governors on LLM Inference Performance and Energy Efficiency

    https://arxiv.org/html/2507.02135v1

  80. Android Thermal API: sustained performance limits

    https://developer.android.com/games/optimize/adpf/thermal

  81. Perfetto PowerRails: accumulated energy and interval calculations

    https://perfetto.dev/docs/reference/trace-packet-proto

  82. MELTing point: Mobile Evaluation of Language Transformers

    https://arxiv.org/html/2403.12844v2

  83. Transformers: why causal KV caching works

    https://huggingface.co/docs/transformers/en/cache_explanation

  84. Android Power Profiler: observations and attribution limits

    https://developer.android.com/studio/profile/power-profiler

  85. Overview of the ML Kit GenAI APIs

    https://developers.google.com/ml-kit/genai

  86. Transformers GenerationConfig: output reservation and termination

    https://huggingface.co/docs/transformers/en/main_classes/text_generation

  87. Gemini Nano on device — Florina Muntenescu & Oli Gaymond, Google DeepMind

    https://www.youtube.com/watch?v=owH1f0N-keY&t=531

  88. Transformers installation: offline mode

    https://huggingface.co/docs/transformers/installation

  89. Android: Build an offline-first app

    https://developer.android.com/topic/architecture/data-layer/offline-first

  90. TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM

    https://www.youtube.com/watch?v=BKWpYIWvAo4&t=1940

  91. Temporal Activity Execution

    https://docs.temporal.io/activity-execution

  92. The On-Device Machine Learning Behind Recorder

    https://research.google/blog/the-on-device-machine-learning-behind-recorder/

  93. Pixel Phone Help: Create, edit & manage transcriptions

    https://support.google.com/pixelphone/answer/16267698?hl=en

  94. W3C Media Capture and Streams

    https://www.w3.org/TR/mediacapture-streams/

  95. Text Embeddings Reveal (Almost) As Much As Text

    https://aclanthology.org/2023.emnlp-main.765.pdf

  96. Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge

    https://www.jhauswald.com/assets/pubs/kang17neuro.pdf

  97. Gemini Nano on device — Florina Muntenescu & Oli Gaymond, Google DeepMind

    https://www.youtube.com/watch?v=owH1f0N-keY&t=362

  98. Firebase AI Logic: hybrid inference on Android

    https://firebase.google.com/docs/ai-logic/hybrid/android/get-started

  99. Scikit-learn: separating fitting, selection, and final evaluation

    https://scikit-learn.org/stable/modules/cross_validation.html

  100. Playwright: Service Workers

    https://playwright.dev/docs/service-workers

  101. MLPerf Tiny Benchmark

    https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/da4fb5c6e93e74d3df8527599fa62642-Paper-round1.pdf

  102. MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases

    https://arxiv.org/html/2406.10290v1

  103. Running Gemma 4 On-Device: 40 Tokens/s on iPhone with MLX

    https://www.youtube.com/watch?v=a2muGkT4WD4&t=490

  104. A Practical Guide to Efficient AI

    https://www.youtube.com/watch?v=Yyg_BoeB2LU&t=751