Why I Still Train Small Models That Fit in a Pocket

It is easy, in a year of Astra and Agents APIs, to decide that every intelligent feature should be a cloud call. I keep coming back to the opposite bet: a small language model or a task-specific chatbot that runs near the user. Sometimes on the phone. Sometimes on a robot’s own boards.

Humanoid groups are making the same choice for a physical reason. A warehouse robot that waits on a round trip to a region cannot recover a dropped tote in time. A focus app that needs the network to classify a notification is already too late. Edge inference is not nostalgia. It is a latency budget.

The case for small, on purpose

• Privacy: a journal, a medical note, or a child’s chat should not need a boarding pass to a data center.
• Cost: usage-based SaaS gets honest when the heavy path is local and the cloud is the overflow.
• Reliability: planes, basements, and shop floors still lose packets.
• Product feel: a 40ms reply is a feature users describe as 'it just knows'.

A working split

I do not pretend a 3B-parameter model will replace a frontier agent on tax law. I give the small model the jobs it can finish: classify, retrieve, summarize, route. I send the long, unfamiliar, or high-risk work upstairs. That is also how I would staff a humanoid: local policy for balance and grasp, a larger planner when the room is new.

let local = try SmallModel.load(named: "router-v3")
let decision = try await local.route(userText)

if decision.needsFrontier {
  return try await cloudAgent.complete(userText, tools: decision.tools)
}
return decision.localReply

The educational point is simple. 2026 made cloud agents excellent and still did not make the radio free, private, or instant. If you care about apps people hold, keep a model they can hold too.

The future of AI is not only in the cloud. A useful slice of it has to live where the hand and the warehouse already are.