[HN Gopher] Show HN: Cactus - Ollama for Smartphones
___________________________________________________________________
Show HN: Cactus - Ollama for Smartphones
Hey HN, Henry and Roman here - we've been building a cross-platform
framework for deploying LLMs, VLMs, Embedding Models and TTS models
locally on smartphones. Ollama enables deploying LLMs models
locally on laptops and edge severs, Cactus enables deploying on
phones. Deploying directly on phones facilitates building AI apps
and agents capable of phone use without breaking privacy, supports
real-time inference with no latency, we have seen personalised RAG
pipelines for users and more. Apple and Google actively went into
local AI models recently with the launch of Apple Foundation
Frameworks and Google AI Edge respectively. However, both are
platform-specific and only support specific models from the
company. To this end, Cactus: - Is available in Flutter, React-
Native & Kotlin Multi-platform for cross-platform developers, since
most apps are built with these today. - Supports any GGUF model
you can find on Huggingface; Qwen, Gemma, Llama, DeepSeek, Phi,
Mistral, SmolLM, SmolVLM, InternVLM, Jan Nano etc. - Accommodates
from FP32 to as low as 2-bit quantized models, for better
efficiency and less device strain. - Have MCP tool-calls to make
them performant, truly helpful (set reminder, gallery search, reply
messages) and more. - Fallback to big cloud models for complex,
constrained or large-context tasks, ensuring robustness and high
availability. It's completely open source. Would love to have more
people try it out and tell us how to make it great! Repo:
https://github.com/cactus-compute/cactus
Author : HenryNdubuaku
Score : 72 points
Date : 2025-07-10 19:20 UTC (3 hours ago)
| max-privatevoid wrote:
| They literally vendored llama.cpp and they STILL called it
| "Ollama for *". Georgi cannot be vindicated hard enough.
| rshemet wrote:
| didn't Ollama vendor Llama cpp too?
|
| Most projects typically start with llama.cpp and then move away
| to proprietary kernels
| ttouch wrote:
| very good project!
|
| can you tell us more about the use cases that you have in mind? I
| saw that you're able to run 1-4B models (which is impressive!)
| rshemet wrote:
| Thank you! it goes without saying that the field is rapidly
| developing, so the use cases range from private AI
| assistant/companion apps to internet connectivity-independent
| copilots to powering private wearables, etc.
|
| We're currently working with a few projects in the space.
|
| For a demo of a familiar chat interface, download
| https://apps.apple.com/gb/app/cactus-chat/id6744444212 or
| https://play.google.com/store/apps/details?id=com.rshemetsub...
|
| For other applications, join the discord and stay tuned! :)
| xnx wrote:
| Is there an .apk for Android?
| rshemet wrote:
| Cactus is a framework - not the app itself. If you're looking
| for an Android demo, you can go to
|
| https://play.google.com/store/apps/details?id=com.rshemetsub...
|
| Otherwise, it's easy to build any of the example apps from the
| repo:
|
| cd react/example && yarn && npx expo run:android
|
| or
|
| cd flutter/example && flutter pub get && flutter run
| felarof wrote:
| This is cool!
|
| We are working on agentic browser (also launched today
| https://news.ycombinator.com/item?id=44523409 :))
|
| Right now we have a desktop version with ollama support, but we
| want to build a mobile chromium fork with local LLM support. Will
| check out cactus!
| rshemet wrote:
| great stuff. (good timing for a post given all the comet news
| too :) )
|
| DM me on BF - let's talk!
| politelemon wrote:
| Very nice, good work. I think you should add the chat app links
| on the readme, so that visitors get a good idea of what the
| framework is capable of.
|
| The performance is quite good, even on CPU.
|
| However I'm now trying it on a pixel, and it's not using GPU if I
| enable it.
|
| I do like this idea as I've been running models in termux until
| now.
|
| Is the plan to make this app something similar to lmstudio for
| phones?
| rshemet wrote:
| appreciate the feedback! Made the demo links more prominent on
| the README.
|
| Some Android models won't support GPU hardware; we'll be
| addressing that as we move to our own kernels.
|
| The app itself is just a demonstration of Cactus performance.
| The underlying framework gives you the tools to build any local
| mobile AI experience you'd like.
| yrcyrc wrote:
| how do i add RAG / personal assistant features on iOS?
| rshemet wrote:
| you can plug in a vector DB and run Cactus embeddings for
| retrieval. Assuming you're using React Native, here's an
| example:
|
| https://github.com/cactus-compute/cactus/tree/main/react#emb...
|
| (Flutter works the same way)
|
| What are you building?
| khalel wrote:
| What do you think about security? I mean, a model with full (or
| partial) access to the smartphone and internet. Even if it runs
| locally, isn't there still a risk that these models could gain
| full access to the internet and the device?
| rshemet wrote:
| The models themselves live in an isolated sandbox. On top of
| that, each mobile app has its own sandbox - isolated from the
| phone's data or tools.
|
| Both the model and the app only have access to the tools or
| data that you choose to give it. If you choose to give the
| model access to web search - sure, it'll have (read-only)
| access to internet data.
| matthewolfe wrote:
| For argument's sake, suppose we live in a world where many high-
| quality models can be run on-device. Is there any concern from
| companies/model developers about exposing their proprietary
| weights to the end user? It's generally not difficult to
| intercept traffic (weights) sent to and app, or just reverse the
| app itself.
| rshemet wrote:
| So far, our focus is on supporting models with fully open-
| sourced weights. Providers who are sensitive about their
| weights typically lock those weights up in their cloud and
| don't run their models locally on consumer devices anyway.
|
| I believe there are some frameworks pioneering model
| encryption, but i think we're a few steps away from wide
| adoption.
| bangaladore wrote:
| Simple answer is they won't send the model to the end user if
| they don't want it used outside their app.
|
| This isn't really anything novel to LLMs of AI models. Part of
| the reason for many previously desktop applications being cloud
| or requiring cloud access is keeping their sensitive IP off the
| end users' device.
| deepdarkforest wrote:
| > _However, both are platform-specific and only support specific
| models from the company_
|
| This is not true, as you are for sure aware. Google AI edge
| supports a lot models, including any Litert model from
| huggingface, pytorch ones etc. [0]. Additionally, it's not even
| platform specific, works for iOS [1].
|
| Why lie? I understand that your framework does more stuff like
| MCP, but I'm sure that's coming for Google's as well. I guess if
| the UX is really better it can work, but i would also say
| Ollama's use cases are quite different because on desktop there's
| a big community of hobbyists that cook up their own little
| pipelines/just chat to LLMs with local models (apart from the
| desktop app devs). But on phones, imo that segment is much
| smaller. App devs are more likely to use the 1st party
| frameworks, rather than 3rd party. I wouldnt even be surprised if
| apple locks down at some points some API's for safety/security
| reasons.
|
| [0]
| https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inf...
|
| [1]
| https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inf...
| rshemet wrote:
| Thanks for the feedback. You're right to point out that Google
| AI Edge is cross-platform and more flexible than our phrasing
| suggested.
|
| The core distinction is in the ecosystem: Google AI Edge runs
| tflite models, whereas Cactus is built for GGUF. This is a
| critical difference for developers who want to use the latest
| open-source models.
|
| One major outcome of this is model availability. New open
| source models are released in GGUF format almost immediately.
| Finding or reliably converting them to tflite is often a pain.
| With Cactus, you can run new GGUF models on the day they drop
| on Huggingface.
|
| Quantization level also plays a role. GGUF has mature support
| for quantization far below 8-bit. This is effectively essential
| for mobile. Sub-8-bit support in TFLite is still highly
| experimental and not broadly applicable.
|
| Last, Cactus excels at CPU inference. While tflite is great,
| its peak performance often relies on specific hardware
| accelerators (GPUs, DSPs). GGUF is designed for exceptional
| performance on standard CPUs, offering a more consistent
| baseline across the wide variety of devices that app developers
| have to support.
| deepdarkforest wrote:
| No worries.
|
| GGUF is more suitable for the latest open-source models, i
| agree there. Quant2/Q4 will probably be critical as well, if
| we don't see a jump in ram. But then again I wonder when/If
| mediapipe will support GGUF as well.
|
| PS, I see you are in the latest YC batch? (below you
| mentioned BF). Good luck and have fun!
| DarmokJalad1701 wrote:
| I would say that while Google's MediaPipe can technically run
| any tflite model, it turned out to be a lot more difficult to
| do in practice with third-party models compared to the
| "officially supported" models like Gemma-3n. I was trying to
| set up a VLM inference pipeline using a SmolVLM model. Even
| after converting it to a tfilte-compatible binary, I struggled
| to get it working and then once it did work, it was super slow
| and was obviously missing some hardware acceleration.
|
| I have not looked at OP's work yet, but if it makes the task
| easier, I would opt for that instead of Google's "MediaPipe"
| API.
| throw777373 wrote:
| Ollama runs on Android just fine via Termux. I use it with 5GB
| models. They even recently added ollama package, there is no
| longer need to compile it from source code.
| rshemet wrote:
| True - but Cactus is not just an app.
|
| We are a dev toolkit to run LLMs cross-platform locally in
| _any_ app you like.
| jadbox wrote:
| How does it work? How does one model on the device get shared
| to many apps? Does each app have it's own inference sdk
| running or is there one inference engine shared to many apps
| (like ollama does). If it's the later, what's the
| communication protocol to the inference engine?
| rshemet wrote:
| Great question. Currently, each app is sandboxed - so each
| model file is downloaded inside each app's sandbox. We're
| working on enabling file sharing across multiple apps so
| you don't have to redownload the model.
|
| With respect to the inference SDK, yes you'll need to
| install the (react native/flutter) framework inside each
| app you're building.
|
| The SDK is very lightweight (our own iOS app is <30MB which
| includes the inference SDK and a ton of other stuff)
| teaearlgraycold wrote:
| Does this download models at runtime? I would have expected a
| different API for that. I understand that you don't want to
| include a multi-gig model in your app. But the mobile flow is
| usually to block functionality with a progress bar on first run.
| Downloading inline doesn't integrate well into that.
|
| You'd want an API for downloading OR pulling from a cache. Return
| an identifier from that and plug it into the inference API.
| rshemet wrote:
| Very good point - we've heard this before.
|
| We're restructuring the model initialization API to point to a
| local file & exposing a separate abstracted download function
| that takes in a URL.
|
| wrt downloading post-install: based on our feedback, this is
| indeed a preferred pattern (as opposed to bundling in large
| files).
|
| We'll update the download API, thanks again.
| teaearlgraycold wrote:
| Sounds good!
| Uehreka wrote:
| This is one hell of an Emperor's New Groove reference, well
| played: https://x.com/filmeastereggs/status/1637412071137759235
| rshemet wrote:
| love this. So many layers deep, we just had a good laugh.
| refulgentis wrote:
| Beware of this, it's a two weeks old project.
|
| Idk who these people are and I am sure they have good intentions,
| but they're wrapping llama.cpp.
|
| That's what "like Ollama" means when you're writing code. That's
| also why there's a ton of comments asking if it's a server or app
| or what (it's a framework that an app would be built to use, you
| can't have an app with a localhost server like ollama on Android
| & iOS)
|
| There's plenty of projects much further ahead, and I don't
| appreciate the amount of times I've seen this project come up in
| conversation the past 24 hours, due to misleading assertions that
| looked LLM-written, and a rush to make marketing claims that are
| just stuff llama.cpp does for you.
| rshemet wrote:
| reminds me of
|
| - "You are, undoubtedly, the worst pirate i have ever heard of"
| - "Ah, but you have heard of me"
|
| Yes, we are indeed a young project. Not two weeks, but a couple
| of months. Welcome to AI, most projects are young :)
|
| Yes, we are wrapping llama.cpp. For now. Ollama too began
| wrapping llama.cpp. That is the mission of open-source software
| - to enable the community to build with each others' projects.
|
| We're enabling the first cross-platform in-app inference
| experience for GGUF models and we're soon shipping our own
| inference kernels fully optimized for mobile to speed up the
| performance. Stay tuned.
|
| PS - we're up to good (source: trust us)
| HenryNdubuaku wrote:
| Please feel free to join our Discord:
| https://discord.com/invite/bNurx3AXTJ
| pj_mukh wrote:
| Amazing, this is so so useful.
|
| Thank you especially for the phone model vs tok/s breakdown. Do
| you have such tables for more models? For models even leaner than
| Gemma3 1B. How low can you go? Say if I wanted to tweak out
| 45toks/s on an iPhone 13?
|
| P.S: Also, I'm assuming the speeds stay consistent with react-
| native vs. flutter etc?
___________________________________________________________________
(page generated 2025-07-10 23:00 UTC)