90 lines
5.5 KiB
Markdown
90 lines
5.5 KiB
Markdown
# Vision Model Benchmark — Android Control via MCP
|
||
**Date:** Aug 13, 2026 · **Bench host:** shadow-death (10.30.20.128, RTX 4080 SUPER 16GB, Ollama 0.32.9)
|
||
**Target:** tiny vision model for GamingPC (10.30.20.186, RTX 3070 8GB) to drive Android via ADB/MCP
|
||
**Device:** Android Cloud VM1301 (Android-x86 9.0, 1024x768, `Android.local:5555`)
|
||
|
||
## Method
|
||
8 test situations captured from a real Android device with **uiautomator XML ground truth**:
|
||
launcher dialog, Chrome welcome, home screen, Settings, Chrome/Wikipedia article, Calculator,
|
||
Settings search, app drawer. Each model ran 4 tasks per situation (identify, describe, **ground
|
||
coordinates**, agent JSON) = 320 scored calls. Grounding score = pixel-distance accuracy against
|
||
the real element center from the XML dump. No vibes — every score is measured.
|
||
|
||
## Results
|
||
|
||
| Model | Size | identify | describe | **ground** | agent JSON | tok/s | VRAM (bench) |
|
||
|---|---|---|---|---|---|---|---|
|
||
| moondream:1.8b | 1.7G | 0.12 | 0.21 | 0.00 | 0.00 | 331 | 1.2G |
|
||
| granite3.2-vision:2b | 1.7G | 0.62 | 0.04 | 0.00 | 0.38 | 223 | 3.1G |
|
||
| **qwen2.5vl:3b** | **2.6G** | **1.00** | **0.55** | **0.57 (0.91 visible)** | **0.62** | **219** | 8.2G* |
|
||
| llava-phi3:3.8b | 2.9G | 0.62 | 0.14 | 0.27 | 0.00 | 196 | 4.0G |
|
||
| gemma3:4b | 3.3G | 1.00 | 0.40 | 0.30 | 0.38 | 161 | 3.0G |
|
||
| llava:7b | 4.7G | 0.62 | 0.10 | 0.03 | 0.62 | 145 | 8.7G |
|
||
| qwen2.5vl:7b | 5.7G | 0.88 | 0.52 | 0.57 (0.91 visible) | 0.38 | 129 | 12.7G |
|
||
| minicpm-v4.5:8b | 6.1G | 1.00 | 0.64 | 0.54 (0.81 visible) | 0.12 | 118 | 11.1G |
|
||
|
||
*bench VRAM includes default 4096 context. On GamingPC with num_ctx=2048: **3.2GB total, fully on GPU.**
|
||
|
||
**Grounding on screens where the target was actually visible** (dialog/chrome/settings/article/search):
|
||
|
||
| Model | avg | per-screen |
|
||
|---|---|---|
|
||
| qwen2.5vl:3b | **0.91** | .90 / .93 / .85 / .94 / .93 |
|
||
| qwen2.5vl:7b | **0.91** | .90 / .94 / .84 / .94 / .93 |
|
||
| minicpm-v4.5 | 0.81 | .50 / .75 / .93 / .90 / .95 |
|
||
| llava-phi3 | 0.43 | inconsistent (0/.32/0/.86/.98) |
|
||
| gemma3:4b | 0.27 | partial only |
|
||
| llava:7b / moondream / granite3.2-vision | ~0 | unusable for tapping |
|
||
|
||
## Winner: qwen2.5vl:3b
|
||
- **Grounding: 0.91** — identical to its 7b big brother at half the size and 40% more speed
|
||
- 1.00 app identification, 219 tok/s, warm answers ~0.5–1.5s
|
||
- On GamingPC 8GB: **3.2GB VRAM** (num_ctx 2048), fits alongside granite4.1:3b keep-warm
|
||
- Live tap test on calculator: 7→(59,310) vs true (73,357) — 20–50px accuracy, hits the buttons
|
||
|
||
## Control-loop demo (GamingPC → VM1301)
|
||
Vision-only loop for "type 7+8 and press =": model tapped **7 → 8 → + → =** in the correct
|
||
sequence with ~20px precision. Multi-step autonomy flaked afterward (pressed back, looped on
|
||
7/8) because a stateless single-screenshot loop can't self-verify. **Fix that works:** keep step
|
||
history in the prompt + let the agent verify with uiautomator (exact bounds) after every tap.
|
||
Production architecture = Hermes agent decides → qwen2.5vl:3b sees/grounds → android-adb MCP
|
||
executes → uiautomator dump verifies.
|
||
|
||
## Model verdicts (the interesting ones)
|
||
- **qwen2.5vl:7b** — no grounding advantage over the 3b. Skip on 8GB cards.
|
||
- **minicpm-v4.5:8b** — best descriptions (0.64) + good grounding (0.81), but 6.1GB, 118 tok/s,
|
||
and its JSON output wanders (multi-action plans, its own schema). Good fallback.
|
||
- **minicpm-v4.6** (1.6GB, on GamingPC) — **thinking-only variant: content is always empty.**
|
||
Unusable for direct control.
|
||
- **granite3.2-vision:2b** — returns *tool calls* (`getpixel`) instead of answers. Wrong fit for
|
||
prompt-based control, interesting for tool-use harnesses.
|
||
- **llava:7b / moondream / llava-phi3** — grounding garbage. Don't bother.
|
||
- **llama3.2-vision:11b** — **fails to load on Ollama 0.32.9** (`unknown model architecture:
|
||
'mllama'`). Not available on this stack.
|
||
|
||
## Pitfalls hit (all fixed)
|
||
1. **Android screen sleep → black screencaps** (4.9KB). Fix: `svc power stayon true`.
|
||
2. **VM1301 IP drifts** — kernel `ip=.26` is overridden by Android's EthernetService DHCP
|
||
(router gave .79). Fix: `adb connect Android.local:5555` (mDNS) — DHCP-proof. android-adb
|
||
MCP now auto-discovers: env → online device → Android.local → legacy .84.
|
||
3. **Stale adb daemon** → "No route to host" despite TCP working. Fix: `adb kill-server`.
|
||
4. **Markdown fences around model JSON** — strict `json.loads` scores 0; fence-tolerant parsing
|
||
recovers 0.0 → 0.62 on agent task.
|
||
5. **qwen2.5vl outputs** either `{"x": int}` or `{"coordinate": [x,y]}` or bbox lists — harness
|
||
must normalize all three.
|
||
6. **urllib routed LAN Ollama through a system proxy** → 404s. Fix: `ProxyHandler({})`.
|
||
7. Realme phone (10.30.20.104) and UGOOS TV (.84) both offline — VM1301 is the live target.
|
||
|
||
## Deployment (done)
|
||
- `qwen2.5vl:3b` + `qwen2.5vl:7b` installed on GamingPC (10.30.20.186:11434)
|
||
- ADB MCP patched for device auto-discovery; verified working against VM1301
|
||
- Recommended call: `num_ctx=2048`, `temperature=0`, ask for `{"coordinate":[x,y]}` JSON
|
||
- Demo harness: `~/vision_bench/android_control_demo.py <model> <goal>`
|
||
- Raw data: `~/vision_bench/bench_results.jsonl` · analysis: `analyze.py`, `final_scores.py`
|
||
|
||
## Recommended follow-ups
|
||
- Pin qwen2.5vl:3b in `keep_ollama_hot.py` on .186 (vision goto, next to ornith)
|
||
- Wire Hermes auxiliary vision to qwen2.5vl:3b@.186 (config.yaml auxiliary.vision)
|
||
- Add step-history + uiautomator verification to the control harness for multi-step tasks
|
||
- Replace dead UGOOS TV .84 default references; keep mDNS path as primary
|