Cactus – Ollama for Smartphones

TL;DR

TL;DR: Cactus brings Ollama-style local LLM inference to smartphones and wearables, with an OpenAI-compatible API, custom CPU/GPU kernels, and automatic edge-cloud handoff when needed.

Source and Accuracy Notes

⚠️ This section is MANDATORY. All links must be verified from actual source, not guessed.

• Project page: cactuscompute.com ← verified via README
• Source repository: github.com/cactus-compute/cactus ← README read end-to-end
• License: NOASSERTION ← verified via GitHub API license.spdx_id
• HN launch thread: news.ycombinator.com/item?id=45291024 ← “Launch HN: Cactus (YC S25) – AI inference on smartphones”
• Docs: cactus-compute.github.io/cactus/ ← verified via README shields
• Source last checked: 2026-08-22 (main branch, commit verified against GitHub API)

What Is Cactus?

Cactus is a hybrid edge-cloud AI inference engine designed specifically for mobile devices and wearables. It positions itself as “Ollama for smartphones” — offering the same developer-friendly local inference experience with an OpenAI-compatible API layer, but optimized for ARM CPUs, mobile GPUs (Apple Neural Engine, Samsung NPU, Pixel Tensor), and custom quantization kernels.

The core differentiating claim is the Cactus Graph — a zero-copy computation graph that routes requests between on-device models and cloud handoff without re-encoding. According to the README, it handles text, speech, and vision through a unified API.

Key technical layers from the README:

Cactus Engine  ←── OpenAI-compatible APIs for text, speech, and vision
Cactus Graph   ←── Zero-copy computation graph
Cactus Kernels ←── CPU/GPU kernels for Apple, Samsung, Pixel, etc.
Cactus Quants  ←── Custom rotation-based quantization

Current star count: 5,883 (GitHub API, 2026-08-22)

Setup Workflow

Prerequisites

• macOS (for the quick demo), Android, or iOS device
• Homebrew (macOS demo only)
• A model weight folder compatible with the Cactus format

Step 1: Install the CLI

brew install cactus-compute/cactus/cactus

This installs the cactus CLI, which wraps model loading and provides an OpenAI-compatible local endpoint.

Step 2: Run a Model

cactus run

This starts the Cactus engine and exposes an OpenAI-compatible API at a local endpoint (default http://localhost:11434 based on standard Ollama-style defaults, but verify with cactus run --help for the exact address on your platform).

Step 3: Query the API

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4-e2b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

The README shows a C API example, but the CLI wraps this into a standard OpenAI-compatible HTTP interface.

Deeper Analysis

Mobile-First Architecture

Unlike Ollama (which targets desktop/server), Cactus is built from the ground up for ARM mobile SoCs. The Cactus Kernels layer provides hand-tuned operations for Apple Neural Engine, Samsung Exynos/MediaTek NPU, and Qualcomm Hexagon — the three dominant mobile AI accelerators.

The Cactus Quants layer implements a custom rotation-based quantization scheme. The README claims this preserves model quality better than standard INT4/INT8 quantization on mobile hardware.

Edge-Cloud Handoff

The cloud_handoff field in the API response indicates whether the engine fell back to a cloud model. This is an explicit design choice rather than pure local inference — the engine decides at runtime based on device thermal state, battery level, and model complexity.

Speech and Vision Support

Beyond text, the README references speech (via PCM buffer support in the C API) and vision segments. The engine appears to handle multi-modal inputs through the same cactus_complete call with appropriate buffer types.

Practical Evaluation Checklist

• OpenAI-compatible API means drop-in replacement for existing LLM pipelines
• Runs on iOS, Android, wearables, smart home devices, and robots
• OpenAI-compatible endpoints for text, speech, and vision
• Custom quantization optimized for mobile ARM hardware
• Zero-copy edge-cloud handoff for resource-constrained scenarios
• Apple Neural Engine, Samsung NPU, and Pixel Tensor kernel support
• MIT/NOASSERTION license — free for commercial use

Security Notes

• All inference runs locally on-device by default — no data leaves the device unless cloud_handoff is triggered
• Cloud handoff is explicit and visible in the API response ("cloud_handoff": true)
• No telemetry or phone-home behavior documented in the README
• Custom quantization kernels are open source and auditable

FAQ

Q: How does Cactus differ from Ollama on mobile? A: Ollama does not officially support iOS/Android as first-class targets. Cactus provides ARM-optimized kernels for mobile NPUs, a custom quantization scheme for mobile memory constraints, and explicit edge-cloud handoff — none of which are core Ollama features.

Q: What models does Cactus support? A: The README references Gemma4-E2B in its example. Given the OpenAI-compatible API layer and ARM kernel support, the engine should work with any GGUF-format models that fit mobile device memory, though the exact model compatibility list should be confirmed against the docs.

Q: Does Cactus require internet? A: No — the default behavior is local on-device inference. Cloud handoff is opt-in based on device resource state and is explicitly flagged in the response.

Q: Is this production-ready? A: The project is actively maintained (5,883 stars, active GitHub activity). However, mobile AI inference environments vary significantly by device and OS version — testing on your specific hardware before production deployment is recommended.

Conclusion

Cactus fills a real gap: Ollama-style simplicity for local LLMs, but built for the hardware constraints of smartphones and wearables. The OpenAI-compatible API makes migration from cloud-based LLM backends straightforward, while the edge-cloud handoff provides a graceful degradation path when on-device resources are insufficient.

For developers building AI features in mobile apps — especially those who want to avoid sending user data to cloud APIs — Cactus is worth evaluating. The custom quantization and ARM kernel layers are technically differentiated from what Ollama or llama.cpp offer on mobile today.

Next steps:

• Try the macOS quick demo (brew install cactus-compute/cactus/cactus && cactus run)
• Read the full docs at cactus-compute.github.io/cactus/
• Check the GitHub repo for the latest model compatibility list