Local AI agents: 14MB running with no internet

Have you ever gotten the AI API bill and thought "but I only used it to sort email"? Right. Most of what companies do with AI is small, repetitive, boring work. And we've been paying rocket prices to go to the corner store. That's why local AI agents, tiny models that run on the device itself, started becoming a serious topic.

A model called Needle2 showed up this week, from Cactus Compute. It takes up 14 megabytes. That's not a typo. Fourteen. A photo on your phone weighs more than that. And it's not a chatbot: it's a model built to call tools, meaning decide which action to run and with which parameters. It runs on phones, watches, smart locks, warehouse robots.

What changes when AI fits on the device

Today the standard is: your app asks, sends it to the cloud, waits, receives, displays. It works fine. But it carries three problems nobody likes facing in a meeting.

First, cost. You pay per token, and token is variable consumption. The more the product succeeds, the more expensive it gets. It's the only business in the world where success punishes you.

Second, latency. A round trip to the cloud costs between 300ms and 2 seconds, depending on the customer's network. For writing text, fine. For a lock deciding whether to open the door, that's an eternity.

Third, data. Every time text leaves your machine, someone has to answer in an audit what happens to it. I've seen a healthcare project stall two months on that question.

A local model solves all three at once. Zero cost per call once the app is installed. Responses in tens of milliseconds. And the data never leaves the device, which makes the privacy conversation a lot shorter.

Not every task needs a genius. Plenty of things just need someone who doesn't hit the wrong button.

But does 14MB think at all?

No, and that's the point. A model that size won't write your sales proposal, won't analyze contracts and won't replace the Claude or GPT you use for real work. If you ask it for market analysis, it will make things up. Confidently.

What it does is one thing, and it does it well: take a sentence and turn it into a structured action.

• "Turn off the living room light" becomes light.off(room="living")
• "Book a meeting with Paulo Tuesday at 2pm" becomes a calendar call with the fields filled in
• "Customer complained about a wrong charge" becomes a ticket with the right category

It's smart routing. Translating human language into machine commands. That's about 70% of what most automations need, and we'd been using billion parameter models to do the job.

The honest comparison: it's the difference between hiring a senior lawyer and hiring a very good receptionist. You don't put the senior on call transfers.

Where this works in a real business

Let me get out of the abstract. Three scenarios that come up here all the time.

Field apps. Maintenance techs, delivery drivers, outside sales reps. People working in warehouses, basements, on the road, places where the mobile signal dies. Today their app is either a tedious form or it depends on internet to "understand" what the person typed. With a local model, the guy says "replaced the compressor, part 4402, customer signed" and the app fills in the report. No signal. It syncs later.

Message triage. A company that gets 300 messages a day on WhatsApp. Classifying each one (quote, support, complaint, spam) via API costs little per message, but it costs. And every message passes through a third party server. A small model running on your own server does the triage, and only the complex stuff goes up to a big model.

Physical devices. Kiosks, turnstiles, control panels, industrial equipment. No debate here: if the decision depends on the cloud and the cloud goes down, the equipment becomes furniture.

Notice the pattern. In none of them is raw intelligence the bottleneck. The bottleneck is cost, connection or privacy.

The trick that's worth more than the model

Here's the part that actually matters, and it holds whether Needle2 exists or not.

The architecture that works is layered. Small model in front, big model behind. The small one handles everything that comes in, solves what's simple and passes along what's hard. It's exactly how good human support works: tier 1 and tier 2.

In practice, in a typical support flow, somewhere between 60% and 80% of what comes in is repetition. Duplicate invoice, business hours, order status, account updates. If the local model handles that range, your API bill drops by the same proportion. And quality doesn't drop, because the hard stuff still goes to the good model.

I've implemented this logic for a client without even using a local model: just a rules filter in front of the AI. The bill dropped 40% in the first month. With a small model in front, the cut is bigger and the filter gets smarter, because it understands spelling variations ("invoice", "invoce", "that payment paper thing").

The lesson isn't "use a tiny model". It's: stop sending everything to the most expensive model out there. That's the AI equivalent of taking a black car service around the corner, every day, and then complaining about the statement.

What I wouldn't do with this yet

A strong opinion, since we're here: I wouldn't put a 14MB model making decisions on its own in anything involving money, health or personal safety. Not because it's bad, but because a small model's error rate is higher, and you need a plan for when it gets things wrong.

The right way to use it is with an escape hatch. If the model isn't sure, it doesn't guess: it escalates. Sends it to the big model, or to a human. That "I don't know", well implemented, is worth more than ten points of accuracy.

It's also not worth embedding this in a product that doesn't exist yet. If you don't have the flow mapped out, putting local AI in first is optimizing something you don't know people will use. First make it work the expensive, simple way. Then make it cheaper with real data in hand.

Where to start without spending

A short path, in order:

1. Take 200 real interactions from your support or your app. Don't make them up, use the real ones.
2. Split them into two piles: "a standard response solves it" and "needs thinking".
3. If the first pile has more than 100 items, you have a clear case. If it has 20, leave it alone and go work on something else.
4. Measure what that first pile costs today: in API, in people's time, in customers waiting.

That number is your budget. If it's small, great, you just saved yourself a project. If it's big, now you know exactly what you're buying.

What's interesting about Needle2 and models like it isn't the size. It's the message baked in: the industry spent two years chasing bigger models and now found out that for a lot of companies the problem is the opposite. There's no shortage of intelligence. What's missing is putting the right intelligence in the right place, at the right cost.

If you have an operation that runs everything through AI and you suspect you're overpaying for simple tasks, this can usually be measured in an afternoon. Tell me what you're automating and we'll see if the savings are real or just nice theory.

Your AI bill went up again and a good chunk of it was just for sorting email.

This week Needle2 showed up, a 14 megabyte model that runs on the device itself, with no internet. It doesn't think, doesn't analyze contracts, doesn't write proposals. It does one thing only: turn a sentence into an action.

And look, about 70% of the automations I see out there need exactly that, but they're running on a billion parameter model.

The math isn't about model size. It's about not taking a black car service around the corner every day and then complaining about the statement.

Small model in front, big model behind, and what it doesn't know it escalates instead of guessing. I've seen costs drop 40% with nothing but a rules filter in front of the AI.

If you suspect you're overpaying for simple tasks, this can be measured in an afternoon. Tell me what you're automating.

#ArtificialIntelligence #LocalAI #EdgeAI #Automation #Technology