LLMs for mobile engineers, from tokens to your app · Puzzle 7 · 8 of 8
On the phone or in the cloud
Puzzle 6 put a server between your app and the model. For some features you can skip the server and the network entirely: Android ships a model that runs on the phone. Here's what it's good at, what it costs you, and how to choose.
The problem
You want "smart reply" suggestions in a messaging app. Sending every private message to a cloud model raises privacy questions, costs money per message, and fails in a tunnel. Is there another way?
See it
An on-device model: Gemini Nano runs inside AICore, a service on the phone itself.
Step 1 of 4A smart reply is made right there: no network, no bill per request, and the message never leaves the phone.
Step 2 of 4A question over a 200-page manual is too much for a small model: it goes through your server to a cloud model. Predict: what about phones without AICore support?
Step 3 of 4They need a plan too: the cloud path, or a simpler feature.
Step 4 of 4
The building block
An on-device model runs on the phone itself. On Android, Google's Gemini Nano runs inside AICore, a system service that uses the device's hardware and keeps the model updated. Apps reach it through ML Kit's GenAI APIs: ready-made tasks like summarising, proofreading and rewriting, and a Prompt API (beta) for your own prompts.
What you gain:
- No network: it works in a tunnel, and there's no wait for a request to reach a server and the answer to come back.
- Privacy: the text never leaves the phone.
- No per-request bill: there's no API to pay for each call.
What you give up:
- Size: a model that fits on a phone is much smaller than a cloud model, so hard reasoning and long documents are out of reach.
- Reach: it runs only on devices that support it, and different devices can run different versions of Gemini Nano. Your feature needs a plan for the rest: a cloud fallback through your backend (puzzle 6), or a simpler feature.
So the decision is per feature. Short, frequent, private tasks the user triggers inside your app, like smart replies or tidying a note, suit the phone. Three limits shape the design: AICore runs the model only while your app is the top foreground app (a foreground service doesn't count); each app has a request quota, a cap on how many requests it may make in a period, so a burst gets a BUSY error to retry with backoff, plus a longer battery quota; and the model may need downloading first. So check the feature's status before you offer it, and show it only when it's ready. Long or hard ones, like answering questions over many documents, go to the cloud, behind your backend.
The analogy
A pocket calculator and a call to an accountant. The calculator is instant, free and private, for everyday sums. For a tax return, you call the accountant, and pay for it.
Check yourself
Sources
- Android Developers: Gemini Nano ("Gemini Nano lets you deliver rich generative AI experiences without needing a network connection or sending data to the cloud. On-device AI is a great solution for use-cases where low cost and privacy safeguards are your primary concerns"; "Gemini Nano runs in Android's AICore system service, which leverages device hardware to enable low inference latency and keeps the model up-to-date")
What you now know
An on-device model like Gemini Nano runs on the phone through Android's AICore service: no network, no per-request bill, and the data never leaves the device. In exchange it's smaller than cloud models and available only on supported devices. Use it for short, private, frequent tasks; use a cloud model behind your backend for hard ones; and design a fallback for devices that can't run it.
Answer every question above correctly to unlock the next puzzle.
That's everything published so far
You've reached the last puzzle of LLMs for mobile engineers, from tokens to your app. Next: Next: what a call costs and how it fails, about max_tokens and a cut-off answer, why output tokens cost more than input, rate limits and retries, prompt caching, and structured output you can parse.
New puzzles are announced on LinkedIn and in the learning log feed.
Something wrong or unclear on this page? Tell me: a puzzle that didn't click is a bug in the puzzle.