▲ 531 points

Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

cactuscompute.com

by HenryNdubuaku·6d ago·182 comments·view on hn ↗

Hey HN,

Henry from Cactus here!

We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2.

The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges 300-700 on sub-$200 phones such as the Samsung A-Series.

On the tool call and mobile device use benchmarks, Needle 2 trades wins with closest small models like LFM2.5 230M and Apple Foundation Model, at 5x to 70x smaller, both at f16 vs Needle 2 at 2bit. Needle is based on Simple Attention Networks from our paper (https://arxiv.org/abs/2607.18363).

Edge AI has lately meant Macs and PCs, but that is just 1.5 billion of over 21 billion connected IoT devices in the world today, and in emerging markets most phones ship under $200, no NPU, cheap GPUs. These include budget phones, Raspberry Pis, microcontrollers, wearables, small robots like Reachy Mini, and connected home devices.

A conventional transformer of Needle's width and depth spends 164 MFLOPs per token, and even one squeezed down to Needle's parameter count spends 87, Needle spends 70. Even on a high-end phone, an always-on assistant lives inside a power budget; every MFLOP is milliwatt-hours, and Needle spends 7x to 85x fewer of them per token than the smallest performant LLMs. More about the architecture in the link.

When we structure intelligence for consumer devices as functions with typed parameters, the only hard part is mapping a messy sentence onto them; which function, with which values. Our research found that when framed that way, the problem needs no world knowledge and no open-ended prose, which is why 45M parameters suffice.

Needle 2 expands to structured extraction where the schema can be passed in-place of tools and the model returns structured output. You can use Needle as a text-classification model with an enum field, as a summarization model by providing a schema that extracts key fields, everything but free-range decode.

Every product has its own tool vocabulary and fine-tuning needle helps it achieve frontier-level performance on custom tasks, so using the python package (https://github.com/cactus-compute/needle), Needle can be fine-tuned Needle on a Mac/PC in minutes to a few hours, with automated data-generation pipeline, just pass a couple samples.

Nonetheless, every response carries a learned confidence score based our Cactus Hybrid technique. If above your threshold, act, below it, escalate to the cloud or bigger model. Combining Needle 2 with a private DeepSeek-v4-Flash deployment works particularly well for enterprise-level tasks at barely any cost, we can help with this setup.

We have put a lot of thoughts into Needle 2 but might still be missing quite a lot, please use the playground in the provided link to test Needle and share your thoughts, always appreciated!

182 comments

This is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack.

With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.

Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).

So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.

> Like, what does a 28MB binary get you?

For one thing, on beefy-enough recent CPUs, you could keep the weights hot in the L2 cache of a single CPU core. (Which is clearly not the use-case, but might be interesting to those looking for extreme TPS numbers. Or perhaps for efficient training!)

I'm quite impressed by the results of the web demo, especially given its size and the precision with which it uses the three available tools (tested with German commands). I could imagine that this LLM would fit well into a setup with multiple micro-sized LLMs for different purposes; so 14 MB for precise tool invocation is a reasonable memory footprint when a number of other local models are running (e.g. STT, TTS + language models).

It's called a LM :). LLM stands for Large Language Model.

It's definitely cool that you can get any reasoning whatsoever out of such a small model. That said, its reasoning is "interesting":

Query: "Make the living room dark" Agent: "User wants lights on in living room. 'dark' implies dim. Room 'living room', action 'on'." (And on every test I did, it just completely ignored the "brightness" parameter)

It also appears to have no concept of what a door or light actually is, whenever the query diverges from "Lock door X" or "Turn on light X", it tries to shoehorn whatever additional context is given into the device name:

Query: "Lock out the vacuum salesman at the front door" Agent tries to lock "front door vacuum salesman"

"The way you talk really makes me appreciate silence" is classified as "positive" with 82% confidence.

Ok, this is genuinely funny, we will fix these as we iterate, thanks lol.

that model is 14MB large what do you expect. but I agree it's funny regardless

My first query:

> Make it a little warmer in here.

The reply:

> "name": "set_thermostat", > "arguments": { > "temperature": 65, > "mode": "cool", > ... > "reasoning": "'warmer' implies need for cooling; set_thermostat with temperature 65 (typical warmth) and mode 'cool'.",

Maybe I'm doing it wrong?

Tested your example, the confidence is 0. In smart home context, I can think of an application where the low confidence answers can be forwarded to cloud, whereas the vast majority generic queries solved locally, if the confidence is reliable enough. The response is quite fast by the way.

Try asking it to set the thermostat to a value. It's a very small model, there's not much reasoning capability

It's not a conversational model. It's meant as a local tool calling model.

Funny result from the web demo. I'm well aware that it's an extremely small and, well, stupid, model, but even so:

Query: HN

Result:

{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.", "confidence": 0 }

I'd expect it to at least ignore (call no tools) for the queries that it doesn't understand. And it seems like it does do that, just not consistently.

That's really cool - I was already thinking of compressing `functiongemma-270m-it` down to 1-2 bits so it would work flawlessly in the browser. Your `Fine-tuning` feature is even much more convenient.

Specifying units seems to be unreliable; I tried adding a description to the set_thermostat temperature:

    "temperature": {
      "type": "number",
      "description": "degrees Fahrenheit"
    },

Set the living room temperature to 70 degrees Celsius

    {
      "function_calls": [
        {
          "name": "set_thermostat",
          "arguments": {
            "room": "living room",
            "temperature": 70,
            "mode": "cool"
          }
        }
      ],
      "confidence": 0.6045
    }

Set the living room temperature to 70 degrees Fahrenheit

    {
      "function_calls": [
        {
          "name": "set_thermostat",
          "arguments": {
            "room": "living room",
            "temperature": 70,
            "mode": "heat"
          }
        }
      ],
      "confidence": 0.4536
    }

Set the living room temperature to 70 degrees

    {
      "function_calls": [
        {
          "name": "set_thermostat",
          "arguments": {
            "room": "living room",
            "temperature": 70
          }
        }
      ],
      "confidence": 0.8517
    }

Trying "in degrees Fahrenheit" for the tool description had similarly counterintuitive confidences.

Edit: to be clear, the counterintuitive behavior is that the confidence ended up higher for the wrong units.

Could someone please share how such open source micro-LLMs might have been created?

Do the creators take something like DeepSeek, and then delete most of the neurons to whittle down the size?

Can you share more about the architectural/design tradeoffs you considered or decided upon? Particularly for me, why is a model that is intended mainly to just make tool calls and marshal the results back focusing on speed? Speed as an inherent result of small size, I get, but speed as a design focus confuses me because it’s simply not going to be dealing with large outputs as a rule, wouldn’t it be better to trade some of that raw speed for better smarts?

For example, I mocked a dumbed down version of what would be a reasonable intermediate tool call prompt:

> It's currently 58 degrees. User asks for house to be 8.5 degrees warmer. What temperature to set thermostat to?

The reply?

Reasoning: “User asks for temperature to set thermostat to 8.5 -> set_thermostat with temperature=8.5.”

Sounds like something Siri would do!

This is cool!

While most of the industry focuses on the frontier of “intelligence” (function), a release like this represents the frontier of the other end of the spectrum (form).

Both are important if we ever want to see “Opus-level” capability running locally on commodity machines in the future.

Nice. I used Needle as a router in https://rcarmo.github.io/projects/memento/, need to take a look at this

Looks incredibly cool, although it ran at ~4 tok/s on my browser (I assume because I've disabled V8). I believe your numbers though. I always wanted to try using something like this to create an AI assistant for my phone. If you try to create something without AI, you'll quickly run into the problem of using NLP to map input to the equivalent of AI tool calls. On the other hand, shelling out to a full-blown LLM for every input is slow, costly and if you want it local, greatly restricts its computing power and the number of devices it can run on. Needle and similar models solve this problem nicely: they're incredibly cheap and handle NLP (not in a predictable way of course, but with great success rates), so you can quickly run predefined actions or shell out to a proper LLM for more complicated tasks. You have the best of two worlds.

You may have inspired me to give it a try!

Congratulations. 28MB is impressive, I've not played around with actual queries/outputs.

I'm wondering what is the overall thesis/plan here and where exactly the innovation lies? Would love if you can throw light on below,

  - If I understand, this is complete stack of a custom architecture (attention only transformers), custom quantisation format and a runtime engine all packaged together?
  - How do you differentiate / compete against LiteRT (former TensorFlowLite) and Lite RT LM? Google is heavily investing in this ecosystem because Android is where they have distribution moat. Wouldn't it be easier for me as a developer to build on top of LiteRT since it is relatively open ecosystem and I can pack large number of open models from HF directly?
  - What exact challenges you saw with TFLite, TVM etc that prompted this effort ?
  - What will be the pricing model like?

This is really cool, I'm curious how much knowledge can their be in smaller models? It seems the current trade off is you need sizeably larger models for more performance but I'm curious if in your work how far this is true, as edge ai is really what needs to get better before physical ai can take off (my two cents).

I don't quite get what this can be used for.

It cannot be used for direct customer interaction, it's simply too limited.

Nobody wants to say "lock the front door", they would just do it

And even quite direct commands like "tighten security" are not understood by the model

Its pretty significant you've got this working locally in wasm. Very cool.

Re: robotics: I'm unsure how this could be helpful.

It fails a pretty simple navigation prompt.

X0: (0.0, 0.0). Object bounding box: [1.0, 1.0, 2.0, 2.0]. navigate to (3.0,3.0)

I changed it to "call path planner to navigate: a_star(x0, xf, obs)"

Another fail.

My intuition tells me micro llms will/are important for robotics. I just can't grok it. Can someone without control theory experience give me a good example?

Probably at the planning level of the navigation stack. That's where I see reasoning being helpful. Lower than that...idk

Give me an example of a robotics prompt that seems useful and I'll give you an example why we don't need LLMs to be a tracking controller, etc.

I tried using it on my x86_64 desktop. Could not use it.

Then I tried using it on a raspberry pi 4. Did not work.

This is specifically for 64 bit arm. I see no reason why you could not enable desktops as well? Just a matter of compilation

When I asked it to call my cable company I got this response. I am wondering how this response comes out… why does it say lights twice and not mention door? LLMs are a mystery to me but I suppose this reasoning is at the heart of the model and it’s only 40% confident in it

  {
  "function_calls": [],
  "reasoning": "No tool for making phone calls or initiating a call. The available tools are limited to smart home lights, thermostat, and lights.",
  "confidence": 0.4096
  }

Makes me think of the demo from some time ago where someone got a ~29M parameter model running on an esp32. I wonder what kind of throughput this could get if a handful of esp32s were strung together...

Edit: I have a pile of d1 minis, but not much time.

This is so cool but for me safety remains my priority first in place. So, if it violates my registered rules, and does what it wasn't supposed too, then what next.

Today AI agents are becoming difficult to safety most especially in critical sectors like banking, a mere mistake may lead to loss of cash or life if it's a hospital

My question is how safe is it for critical sectors

> set lights on

{ "function_calls": [ { "name": "set_lights", "arguments": { "room": "bedroom", "state": "on" } } ], "confidence": 0 }

is confidence 0 that its confident?

Curious, why did you go down to 2bit rather than 4 bits? 4bit with folding the layers should arrive at the same size, but with better quality?

This is great, I have tried a couple different ways to hook up my Home assistant with Openclaw, Maybe my claw is bloated but a custom agent implementation which just has access to HA tool calling performs way better, although it costs money(Claude API). Would love to give this model a try and see how it performs.

Congrats on this release. The WASM implementation is really cool. This is a surprisingly good fit for a lot of cases, and I totally want to try turning this into a helper assistant for an application.

Please, though, take a pass at humanizing the text on the page. It's Clauded up all over and makes it hard to read.

If you want Needle2 to rget lots of testing, become well known, etc - make a Home Assistant plugin.

What about use case for replacing regex? I.e "random formatted title.extension" - extract the title or some tag or something for more dynamic string manipulation for pulling structured data out of strings efficiently and more simply than regex provides?

> turn on the tv

{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "tv" } } ], "confidence": 0.0158 }

Very interesting, seems confidence is 0 when tool calls are right?

> Turn the lights down low in the bedroom

Sets lights to 30% but also off

> Turn the lights low in the bedroom

Sets lights to on

This is a cool idea but I think humans assume more than 14MB of intelligence. This is like the unhelpful guard in the swamp castle of Monty Python's Holy Grail

This is very interesting! I'm going to spend some time with this. This is really the only class of LLM I'm interested in at all. I sincerely hope on-device takes over and everyone looses their asses on these data centers.

Naïve and clumsy question: how would you pair this with speech-text-speech stuff, wake words etc.? Are there good examples of this for a Pi 5?

The demo is super — I'm just having trouble seeing the whole picture for e.g. a screenless device.

ETA: pun not intended

Was really cool to see yous use Engrams to cut down compute!

Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you've tried ablating engram layers and sizes across your setup?

Also, why mHC over attention residuals?

Any instructions available for running this on an ESP32-S3 or P4 like the site says?

I wonder if there's any way to get this to plan out a dag of tool calls? i.e. use the results from earlier calls as the parameters to later ones? I tried introducing a stack based system, but gave up pretty quickly.

Utterly Fascinating!

For the longest time, I conceptualized LLM's as Text Input -> Text Output transformers, then later as Text Input -> Video Output transformers. Later still I conceptualized them (if they were general purpose) as Any Format Input -> Any Format Output transformers...

The idea of a smaller parameter model runable on smaller/slower/less complex hardware (computers with no GPU, slower CPU's, less memory, aka "Edge Devices") trained for Text Input -> JSON Output (used for tool calls, etc.) I could honestly not conceptualize before seeing the demo on the web page...

But now that I've seen it and conceptualized it -- I'd have to say: "Yes, there's definitely a huge niche, a huge market for this, directly between the non-LLM driven tools and software and SaaS's of yesteryear, and the latest, cutting edge Frontier AI models of today!"

So, I like Needle a lot!

I like Needle a lot, and I love the idea of any tiny resource-thrifty LLM that can run on older hardware, that outputs only JSON!

I can see a huge market for it!

I see the reasoning is always last, but I have seen models do better if you put the reasoning as the first json fileld

Looking forward to npm version of needle-rs supporting v2. I added needle support for tool use in my side project.

What’s one of the most impressive usecase or example?

these micro LLMs could be a game changer for hearing aids

so many interesting lowfi hardware use cases

I’d be interested in attempting to run this in a network- enabled 32MiB RAM microVM.

What is the difference between this and random sentence generator?