Run AI Models Locally on Android with Termux

July 10, 2026 · 11 min read

Why Run AI Locally?

Cloud AI is powerful, but it costs money, requires internet, and sends your data to someone else's servers. Running models locally on your Android phone gives you:

• Zero cost — No API fees, no subscriptions
• Privacy — Everything stays on your device
• Offline access — Works without internet
• No rate limits — Run as much as you want

And modern Android phones (8GB+ RAM) can run surprisingly capable models.

What You Need

• Android phone with 6GB+ RAM (8GB recommended)
• Termux from F-Droid (not Play Store version)
• At least 4GB free storage space
• Patience — setup takes 20-30 minutes

1. Install Termux & Dependencies

pkg update && pkg upgrade -y
pkg install -y git cmake python ninja build-essential

2. Install llama.cpp (CPU + Vulkan)

llama.cpp is the most efficient LLM runner for Android. The Vulkan build uses your GPU for faster inference.

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON
cmake --build . --config Release -j$(nproc)

Build time: 10-20 minutes depending on your phone.

3. Download a Model

For 8GB RAM phones, use Q4_K_M quantized models (4-bit, ~4-6GB):

cd ~
mkdir models && cd models
# Download Llama 3.2 3B (good all-rounder, ~2GB)
wget https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf

# Or Qwen 2.5 Coder 1.5B (fast coding, ~1GB)
wget https://huggingface.co/bartowski/Qwen2.5-Coder-1.5B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-1.5B-Instruct-Q4_K_M.gguf

4. Run Your First Model

cd ~/llama.cpp/build
./bin/llama-cli \
  -m ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  -p "Explain quantum computing in simple terms" \
  -n 512 \
  -t 4

Expect 5-15 tokens/second on a Snapdragon 8 Gen 2 or equivalent.

5. Chat Mode (Interactive)

./bin/llama-cli \
  -m ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  -p "You are a helpful assistant." \
  -n 2048 \
  -t 4 \
  --chat-template llama3 \
  --interactive

6. Server Mode (API)

Run llama.cpp as an HTTP server for other tools to connect:

./bin/llama-server \
  -m ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080

Then curl from another Termux session:

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":"Hello!"}]}'

7. Image Generation with Stable Diffusion

For image generation, use stable-diffusion.cpp:

git clone https://github.com/leejet/stable-diffusion.cpp
cd stable-diffusion.cpp
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DSD_VULKAN=ON
cmake --build . --config Release -j$(nproc)

Download a model and generate:

wget https://huggingface.co/swl-models/rundiffusion-v3/resolve/main/rundiffusion-v3-Q4_K_M.gguf
./bin/sd -m rundiffusion-v3-Q4_K_M.gguf -p "a cute cat, anime style"

8. Speech-to-Text with Whisper.cpp

git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
bash models/download-ggml-model.sh base
make -j$(nproc)
./main -m models/ggml-base.bin -f samples/jfk.wav

Performance Tips

• Use -t 4 for CPU threads (most phones have 8 cores, reserve 4 for system)
• Close other apps to free RAM
• Keep your phone plugged in — inference drains battery fast
• Use a cooling pad for sustained performance

Integrated with 9Router

You can point 9Router at your local llama.cpp server as a custom provider. Add this to your 9Router config:

providers:
  - name: local
    base_url: http://localhost:8080
    models:
      - local/Llama-3.2-3B

Now use local/Llama-3.2-3B alongside all your cloud providers through the same API.

Bottom Line

Running AI locally on Android is not a gimmick — it's genuinely useful for private, offline AI tasks. Llama 3.2 3B handles summarization, brainstorming, and coding help well. For heavy lifting (70B models), you still need cloud. But for everyday AI, your phone is enough.