back
▲ 231 points

Show HN: Cactus – Ollama for Smartphones

github.com
by HenryNdubuaku·1y ago·82 comments·view on hn ↗
Hey HN, Henry and Roman here - we've been building a cross-platform framework for deploying LLMs, VLMs, Embedding Models and TTS models locally on smartphones.

Ollama enables deploying LLMs models locally on laptops and edge severs, Cactus enables deploying on phones. Deploying directly on phones facilitates building AI apps and agents capable of phone use without breaking privacy, supports real-time inference with no latency, we have seen personalised RAG pipelines for users and more.

Apple and Google actively went into local AI models recently with the launch of Apple Foundation Frameworks and Google AI Edge respectively. However, both are platform-specific and only support specific models from the company. To this end, Cactus:

- Is available in Flutter, React-Native & Kotlin Multi-platform for cross-platform developers, since most apps are built with these today.

- Supports any GGUF model you can find on Huggingface; Qwen, Gemma, Llama, DeepSeek, Phi, Mistral, SmolLM, SmolVLM, InternVLM, Jan Nano etc.

- Accommodates from FP32 to as low as 2-bit quantized models, for better efficiency and less device strain.

- Have MCP tool-calls to make them performant, truly helpful (set reminder, gallery search, reply messages) and more.

- Fallback to big cloud models for complex, constrained or large-context tasks, ensuring robustness and high availability.

It's completely open source. Would love to have more people try it out and tell us how to make it great!

Repo: https://github.com/cactus-compute/cactus

82 comments
This is one hell of an Emperor’s New Groove reference, well played: https://x.com/filmeastereggs/status/1637412071137759235
love this. So many layers deep, we just had a good laugh.
“ Is available in Flutter, React-Native & Kotlin Multi-platform for cross-platform developers, since most apps are built with these today.”

Is this really true? Where are these stats coming from?

Probably they mean new apps. Since kotlin multiplatform on android is just native android and android share is like 70% if devices it already at least 50% market share of mobile apps. If you add flutter and react native there is not much left: only games like unity and unreal. I see much less iOS jobs these days.
Ollama runs on Android just fine via Termux. I use it with 5GB models. They even recently added ollama package, there is no longer need to compile it from source code.
True - but Cactus is not just an app.

We are a dev toolkit to run LLMs cross-platform locally in any app you like.

Didn't know that. Thanks
Amazing, this is so so useful.

Thank you especially for the phone model vs tok/s breakdown. Do you have such tables for more models? For models even leaner than Gemma3 1B. How low can you go? Say if I wanted to tweak out 45toks/s on an iPhone 13?

P.S: Also, I'm assuming the speeds stay consistent with react-native vs. flutter etc?

thank you! We're continue to add performance metrics as more data comes in.

A Qwen 2.5 500M will get you to ≈45tok/sec on an iPhone 13. Inference speeds are somewhat linearly inversely proportional to model sizes.

Yes, speeds are consistent across frameworks, although (and don't quote me on this), I believe React Native is slightly slower because it interfaces with the C++ engine through a set of bridges.

This is great!

It would be great if the local llm have access to local tools you can enable/disable as needed (e.g. via customizable profiles). Simple tools such as fetch url, file access, messaging, calendar, etc would be very useful, though I'm not sure if the input token limit is large enough to allow this. Even better if it can somehow do web search but I understand it would be hard to do for free.

Also, how cool it would be if you can expose openai compatible api that can be accessed from other devices in your local network? Imagine turning your old phones into local llm servers. That would be very cool.

By the way, I can't figure out how to clear previous chats data. Is it hidden somewhere?

no, good observation - not hidden; we don't have a "clear conversation" button.

to your previous point - Cactus fully supports tool calling (for models that have been instruction-trained accordingly, e.g. Qwen 1.7B)

for "turning your old phones into local llm servers", Cactus is likely not the best tool. We'd recommend something like actual Ollama or Exo

What do you think about security? I mean, a model with full (or partial) access to the smartphone and internet. Even if it runs locally, isn't there still a risk that these models could gain full access to the internet and the device?
The models themselves live in an isolated sandbox. On top of that, each mobile app has its own sandbox - isolated from the phone's data or tools.

Both the model and the app only have access to the tools or data that you choose to give it. If you choose to give the model access to web search - sure, it'll have (read-only) access to internet data.

Is this using only llama.cpp as inference engine? How is this days support there on NPU and GPU? Not sure if LLM can run on NPU but many models like STT and TTS and vision often can run much faster on Apple NPU
You don't need to guess: https://github.com/cactus-compute/cactus/tree/main/cpp
FYI I see you have SmolLM2, this was replaced with SmolLM 3 this week!

Would be great to have a few larger models to choose from too, Qwen 3 4b, 8b etc

in the app you mean?

Adding shortly!

This is cool!

We are working on agentic browser (also launched today https://news.ycombinator.com/item?id=44523409 :))

Right now we have a desktop version with ollama support, but we want to build a mobile chromium fork with local LLM support. Will check out cactus!

great stuff. (good timing for a post given all the comet news too :) )

DM me on BF - let's talk!

Does this download models at runtime? I would have expected a different API for that. I understand that you don’t want to include a multi-gig model in your app. But the mobile flow is usually to block functionality with a progress bar on first run. Downloading inline doesn’t integrate well into that.

You’d want an API for downloading OR pulling from a cache. Return an identifier from that and plug it into the inference API.

Very good point - we've heard this before.

We're restructuring the model initialization API to point to a local file & exposing a separate abstracted download function that takes in a URL.

wrt downloading post-install: based on our feedback, this is indeed a preferred pattern (as opposed to bundling in large files).

We'll update the download API, thanks again.

This is actually crazy. The API is so simple! I tried to do this on Swift using LLM.swift and it went okay, excited to try this on RN
looking forward to your feedback!
This is a cool app! I'm happy to play with it. Some feedback:

1. The lack of a dark mode is an accessibility issue for me. I have a genetic condition that causes severe light sensitivity and special difficulty with contrast. The only way for me to achieve sufficient contrast without uncomfortable and blinding brightness is dark mode, so at present I can only use your app by disabling dark mode and inverting colors across my phone. This is of course not ideal because it ruins photos in other apps, and I end up with some unavoidable very bright white "hotspots" on my phone that I don't normally have when I can just use dark mode. Relatedly, the contrast for some of the text in the app is low to the point of being practically unreadable for me (getting enough contrast with it similarly requires cranking up the brightness). :(

2. I tried downloading a few other models, namely Jan Nano and SmolLM3, using the GGUF link download functionality, but every time I select them, the app just immediately crashes.

I understand that the chat app on the Play Store is basically just a demo for the framework, and if I were really using it I would be in charge of my own theming and downloading the required models and so on, but these still seem worth fixing to me.

very good project!

can you tell us more about the use cases that you have in mind? I saw that you're able to run 1-4B models (which is impressive!)

Thank you! it goes without saying that the field is rapidly developing, so the use cases range from private AI assistant/companion apps to internet connectivity-independent copilots to powering private wearables, etc.

We're currently working with a few projects in the space.

For a demo of a familiar chat interface, download https://apps.apple.com/gb/app/cactus-chat/id6744444212 or https://play.google.com/store/apps/details?id=com.rshemetsub...

For other applications, join the discord and stay tuned! :)

Very nice, good work. I think you should add the chat app links on the readme, so that visitors get a good idea of what the framework is capable of.

The performance is quite good, even on CPU.

However I'm now trying it on a pixel, and it's not using GPU if I enable it.

I do like this idea as I've been running models in termux until now.

Is the plan to make this app something similar to lmstudio for phones?

appreciate the feedback! Made the demo links more prominent on the README.

Some Android models won't support GPU hardware; we'll be addressing that as we move to our own kernels.

The app itself is just a demonstration of Cactus performance. The underlying framework gives you the tools to build any local mobile AI experience you'd like.

For argument's sake, suppose we live in a world where many high-quality models can be run on-device. Is there any concern from companies/model developers about exposing their proprietary weights to the end user? It's generally not difficult to intercept traffic (weights) sent to and app, or just reverse the app itself.
I understand this is targetted towarda devwlopers, bur Can someone explain why should I go through this complex install process instead of just using ChatterUI? It can handle same GGUF format and works great with Gemma and Qwen. What kind of usecases am I missing?
Love having this running on my Android phone. Reckon i am going to use to automate some task reminders i can do during my commute!
GGUF is easy to implement, but you'd probably find better performance with tflite on mobile for their custom XNNPACK kernels. Performance is pretty critical on low-power devices.
Please feel free to join our Discord: https://discord.com/invite/bNurx3AXTJ
Very cool. Looks like it might be practical to run 7b models at Q4 on my phone, That would make it truly useful!
appreciate if you can provide a apk that does not require google play services to run...
Do the community tools in Ollama work in Cactus? (Just python scripts I think).
They literally vendored llama.cpp and they STILL called it "Ollama for *". Georgi cannot be vindicated hard enough.
I've installed the Android version from https://play.google.com/store/apps/details?id=com.rshemetsub...

It is fantastic. Compared to another program I had installed a year ago, the speed of processing and answering is really good and accurate. Was able to ask mathematical questions, basic translation between different languages and even trivia about movies released almost 30 years ago.

Things to improve: 1) sometimes the question would get stuck on the last phrase and keep repeating it without end. 2) The chat does not scroll the window to follow the answer and we have to scroll manually.

In either case, excellent start. It is without the fastest offline LLM that I've seen working on this phone.

Great project! I will try it out. :)
Does this support openrouter?
how do i add RAG / personal assistant features on iOS?
Is there an .apk for Android?
[flagged]
exciting
Running LLMs, VLMs, and TTS models locally on smartphones is quietly redefining what 'edge AI' means suddenly, the edge is in your pocket, not just at the network boundary. The next wave of apps will be built by those who treat mobile as the new AI server
[flagged]