Snapshot: full project state

REPORT.md

@@ -0,0 +1,89 @@
+# Vision Model Benchmark — Android Control via MCP
+**Date:** Aug 13, 2026 · **Bench host:** shadow-death (10.30.20.128, RTX 4080 SUPER 16GB, Ollama 0.32.9)
+**Target:** tiny vision model for GamingPC (10.30.20.186, RTX 3070 8GB) to drive Android via ADB/MCP
+**Device:** Android Cloud VM1301 (Android-x86 9.0, 1024x768, `Android.local:5555`)
+
+## Method
+8 test situations captured from a real Android device with **uiautomator XML ground truth**:
+launcher dialog, Chrome welcome, home screen, Settings, Chrome/Wikipedia article, Calculator,
+Settings search, app drawer. Each model ran 4 tasks per situation (identify, describe, **ground
+coordinates**, agent JSON) = 320 scored calls. Grounding score = pixel-distance accuracy against
+the real element center from the XML dump. No vibes — every score is measured.
+
+## Results
+
+| Model | Size | identify | describe | **ground** | agent JSON | tok/s | VRAM (bench) |
+|---|---|---|---|---|---|---|---|
+| moondream:1.8b | 1.7G | 0.12 | 0.21 | 0.00 | 0.00 | 331 | 1.2G |
+| granite3.2-vision:2b | 1.7G | 0.62 | 0.04 | 0.00 | 0.38 | 223 | 3.1G |
+| **qwen2.5vl:3b** | **2.6G** | **1.00** | **0.55** | **0.57 (0.91 visible)** | **0.62** | **219** | 8.2G* |
+| llava-phi3:3.8b | 2.9G | 0.62 | 0.14 | 0.27 | 0.00 | 196 | 4.0G |
+| gemma3:4b | 3.3G | 1.00 | 0.40 | 0.30 | 0.38 | 161 | 3.0G |
+| llava:7b | 4.7G | 0.62 | 0.10 | 0.03 | 0.62 | 145 | 8.7G |
+| qwen2.5vl:7b | 5.7G | 0.88 | 0.52 | 0.57 (0.91 visible) | 0.38 | 129 | 12.7G |
+| minicpm-v4.5:8b | 6.1G | 1.00 | 0.64 | 0.54 (0.81 visible) | 0.12 | 118 | 11.1G |
+
+*bench VRAM includes default 4096 context. On GamingPC with num_ctx=2048: **3.2GB total, fully on GPU.**
+
+**Grounding on screens where the target was actually visible** (dialog/chrome/settings/article/search):
+
+| Model | avg | per-screen |
+|---|---|---|
+| qwen2.5vl:3b | **0.91** | .90 / .93 / .85 / .94 / .93 |
+| qwen2.5vl:7b | **0.91** | .90 / .94 / .84 / .94 / .93 |
+| minicpm-v4.5 | 0.81 | .50 / .75 / .93 / .90 / .95 |
+| llava-phi3 | 0.43 | inconsistent (0/.32/0/.86/.98) |
+| gemma3:4b | 0.27 | partial only |
+| llava:7b / moondream / granite3.2-vision | ~0 | unusable for tapping |
+
+## Winner: qwen2.5vl:3b
+- **Grounding: 0.91** — identical to its 7b big brother at half the size and 40% more speed
+- 1.00 app identification, 219 tok/s, warm answers ~0.5–1.5s
+- On GamingPC 8GB: **3.2GB VRAM** (num_ctx 2048), fits alongside granite4.1:3b keep-warm
+- Live tap test on calculator: 7→(59,310) vs true (73,357) — 20–50px accuracy, hits the buttons
+
+## Control-loop demo (GamingPC → VM1301)
+Vision-only loop for "type 7+8 and press =": model tapped **7 → 8 → + → =** in the correct
+sequence with ~20px precision. Multi-step autonomy flaked afterward (pressed back, looped on
+7/8) because a stateless single-screenshot loop can't self-verify. **Fix that works:** keep step
+history in the prompt + let the agent verify with uiautomator (exact bounds) after every tap.
+Production architecture = Hermes agent decides → qwen2.5vl:3b sees/grounds → android-adb MCP
+executes → uiautomator dump verifies.
+
+## Model verdicts (the interesting ones)
+- **qwen2.5vl:7b** — no grounding advantage over the 3b. Skip on 8GB cards.
+- **minicpm-v4.5:8b** — best descriptions (0.64) + good grounding (0.81), but 6.1GB, 118 tok/s,
+  and its JSON output wanders (multi-action plans, its own schema). Good fallback.
+- **minicpm-v4.6** (1.6GB, on GamingPC) — **thinking-only variant: content is always empty.**
+  Unusable for direct control.
+- **granite3.2-vision:2b** — returns *tool calls* (`getpixel`) instead of answers. Wrong fit for
+  prompt-based control, interesting for tool-use harnesses.
+- **llava:7b / moondream / llava-phi3** — grounding garbage. Don't bother.
+- **llama3.2-vision:11b** — **fails to load on Ollama 0.32.9** (`unknown model architecture:
+  'mllama'`). Not available on this stack.
+
+## Pitfalls hit (all fixed)
+1. **Android screen sleep → black screencaps** (4.9KB). Fix: `svc power stayon true`.
+2. **VM1301 IP drifts** — kernel `ip=.26` is overridden by Android's EthernetService DHCP
+   (router gave .79). Fix: `adb connect Android.local:5555` (mDNS) — DHCP-proof. android-adb
+   MCP now auto-discovers: env → online device → Android.local → legacy .84.
+3. **Stale adb daemon** → "No route to host" despite TCP working. Fix: `adb kill-server`.
+4. **Markdown fences around model JSON** — strict `json.loads` scores 0; fence-tolerant parsing
+   recovers 0.0 → 0.62 on agent task.
+5. **qwen2.5vl outputs** either `{"x": int}` or `{"coordinate": [x,y]}` or bbox lists — harness
+   must normalize all three.
+6. **urllib routed LAN Ollama through a system proxy** → 404s. Fix: `ProxyHandler({})`.
+7. Realme phone (10.30.20.104) and UGOOS TV (.84) both offline — VM1301 is the live target.
+
+## Deployment (done)
+- `qwen2.5vl:3b` + `qwen2.5vl:7b` installed on GamingPC (10.30.20.186:11434)
+- ADB MCP patched for device auto-discovery; verified working against VM1301
+- Recommended call: `num_ctx=2048`, `temperature=0`, ask for `{"coordinate":[x,y]}` JSON
+- Demo harness: `~/vision_bench/android_control_demo.py <model> <goal>`
+- Raw data: `~/vision_bench/bench_results.jsonl` · analysis: `analyze.py`, `final_scores.py`
+
+## Recommended follow-ups
+- Pin qwen2.5vl:3b in `keep_ollama_hot.py` on .186 (vision goto, next to ornith)
+- Wire Hermes auxiliary vision to qwen2.5vl:3b@.186 (config.yaml auxiliary.vision)
+- Add step-history + uiautomator verification to the control harness for multi-step tasks
+- Replace dead UGOOS TV .84 default references; keep mDNS path as primary