Add TESTING_GAPS.md audit and link from tests README.

Document honest coverage limits (live AWS, spread, SSRF, missing E2E routes) so operators know what -SkipE2E does not prove.

docs/TESTING_GAPS.md

@@ -0,0 +1,163 @@
+# Test coverage gaps
+
+Operator-focused audit of what automation **does** and **does not** prove. Counts (2026-06-07): Go server **1007** `Test*`, agent **680**, Vitest **867**, Playwright **32** — see [`tests/README.md`](../tests/README.md). Gates and rungs: [`VALIDATION_LADDER.md`](VALIDATION_LADDER.md). By-design limits and manual work: [`PROBLEMS.md`](../PROBLEMS.md).
+
+**Headline:** CI proves wiring, contracts, and mock-heavy paths. It does **not** prove live AWS, real spread lanes, SSRF canary confirmation, LAN subnet discovery, live forge, Android runtime, or fleet scale in your environment.
+
+**Default full gate** (`scripts/test-suite.ps1` phases 1–8) runs all Go, Vitest, and Playwright. **Quick verify** (`-SkipE2E`) drops **all 32 Playwright tests** — common post-landing smoke, easy to mistake for “release green.” Feature slices (`-S3Swarm`, `-CloudMap`, etc.) are focused subsets, not extra coverage beyond `go test ./...`.
+
+---
+
+## 1. Critical gaps — shipped, little or no real automation
+
+| Area | What exists | What's missing |
+|------|-------------|----------------|
+| **Live AWS** | Mock `HeadBucket`/inject S3 store (`s3_swarm_test.go`); ZIP exports for SSM, Fargate, policy fan-out; Emberwake Vitest for method catalog | Real `PutObject`, CloudFront signed magnets, `SendCommand`, `RunTask`, operator Lambda/EventBridge (`PROBLEMS.md` § AWS) |
+| **Cloud Map `route_via` on deploy plans** | Agent registry fetch + `route_via=` in join notes (`cloud_map_test.go`) | Server `AttachCloudMapRouteVia` **deferred** — no wiring tests (`PROBLEMS.md`) |
+| **P2 spread lanes** | Template export, lane dispatch, injectable staging (`staging_chain_test.go`, `spread_lanes_test.go`) | Live Docker/Podman, WinRM/GPO/systemd/crontab on remote hosts, live BITS/curl (`PROBLEMS.md` manual only) |
+| **Deploy Recon SSRF canary** | Go flow with **injected** dial/fetch hooks + manual `CanaryPing` (`recon_handler_test.go`); Playwright **mocks** `POST /recon/scan` (`deploy-recon.spec.ts`) | No automated “paste probe URL into owned target → server fetch → canary confirmed” (`PROBLEMS.md`) |
+| **Subnet recon (live LAN)** | `-SubnetRecon` gate: policy ingest, capped targets, mock port probes (`subnet_recon_test.go`, `subnet_discovery_test.go`) | Real ARP/subnet sweep on LAN; `service_discover` **skips** live LAN in agent tests (`aggressive_commands_test.go`) |
+| **Live HTTP forge** | Builder unit tests | `LIVE_FORGE=1` + `-tags liveforge` only (`live_forge_smoke_test.go`); `e2e-validate.ps1 -ForgeAgent` manual |
+| **Fleet AI / Court LLM** | `DecideFunc` inject, deterministic contingency persona (`persona_mining_test.go`, `contingency_tree_test.go`) | Live Ollama/vLLM; court on **every** exhaust tick (`PROBLEMS.md`, `tests/README.md` Fleet AI gaps) |
+| **Android agent** | APK builder/config/scout Go tests; `android/forge/config_test.go` | No emulator/device E2E; no Crucible round-trip for `platform=android` |
+| **Fusion module** | Phase 3: `go test` + `go build` in `fusion/` | Only `media_crypto_test.go` (decrypt round-trip) — no end-to-end fusion bundle + agent launch |
+| **Mesh P2P** | Stub tests default build (`mesh_p2p_stub_test.go`) | Real peers need `-tags p2p` (`PROBLEMS.md`) |
+| **Scale / soak** | Some `stats_batch` coalesce tests | No 500+/1000+ agent soak, `SQLITE_BUSY`, WS init blob at fleet scale (`PROBLEMS.md` scale limits) |
+| **Activity Feed + ROI** | Routes in `App.tsx`, nav in `Layout.tsx` | **No Vitest, no Playwright** — only TSX pages |
+
+**Stale doc note:** `PROBLEMS.md` still says Path Tracer sessions are RAM-only. Code now has SQLite persist + startup restore (`pathtracer_handler.go`, `pathtracer_persist_test.go`). Gap is **full process restart E2E** and **live WireGuard chain**, not “no persistence.”
+
+---
+
+## 2. Partial coverage — mocks/stubs, not real behavior
+
+### Spread / mining
+
+- WinRM/GPO/Linux LOTL: script **shape** and deploy-plan rendering; execution is not run against domain/SSH targets.
+- Container mining: `SetRuntimeDetector` inject; no live `docker run` / GPU passthrough.
+- `mining_diagnostics` over WS: agent integration uses `stubFastMiningDiagnostics` (~1s+ probes avoided).
+- Discover→spread Playwright: POST `discover_and_join` + stub WS `join_lane: dns_txt` — **UI wiring only** (`discover-spread.spec.ts`).
+
+### Cloud lanes
+
+Each has a `-Feature` slice in `test-suite.ps1`; all mock/export in CI:
+
+- **S3Swarm:** mock shard store, not real bucket policy + signing keys.
+- **SSM:** document + CLI export; no `SendCommand`.
+- **Fargate:** task-def ZIP + campaign sync; no ECS/Fargate in VPC.
+- **PolicyFanout:** public snapshot URL + fan-out ZIP; relay is operator-deployed.
+- **CloudVenue:** IMDS tag inference via inject; not real `g4dn`/spot labels on EC2.
+
+### Contingency / court
+
+- Agent `ContingencyTreeRunner` unit-tested (branch order, ghost skip).
+- Server pushes `contingency_policy` when AI control on (`contingency_bridge_test.go`).
+- Court UI: Vitest + Playwright with **mocked** `GET /ai/decisions` (`lotl-timeline.spec.ts`) — not a live stuck-host → scheduler → LLM → L4 dispatch loop.
+
+### Path Tracer
+
+- Handler-restart persistence unit test.
+- Playwright **mocks** `/pathtrace/start` and `/status` (`path-tracer.spec.ts`); no `wg_setup` E2E (agent test skips UPnP: `aggressive_commands_test.go`).
+- Non-Windows agents: `pathtracer_stub.go` errors on `wg_setup`.
+
+### Deploy Recon
+
+- Port dial and HTTP crawl run on **dashboard host** (Go server), tested via hooks — not through fleet agents.
+- Playwright never hits a real scan; SSRF finding is from **fixture JSON**.
+
+### Subnet recon
+
+- Logic for skip fleet IP, cap hosts, WS ingest — not a lab with multiple live subnets.
+
+### Fusion
+
+- Compile + crypto helper only; fusion zip → dropper → agent path is handler unit tests, not a fused-artifact smoke.
+
+---
+
+## 3. Missing E2E — nav routes vs Playwright (10 specs)
+
+**Layout nav** (`Layout.tsx` `NAV_BASE` + conditional Seer): Command Deck, Crucible, Activity, ROI, Onion, Path Tracer, Forge, Mission Deck, Builds, Emberwake, Deploy Recon, Calibrate; Seer when `ai_control_enabled`.
+
+| Route | Vitest | Playwright |
+|-------|--------|------------|
+| `/dashboard` | yes | `pages.spec.ts`, `smoke.spec.ts` |
+| `/crucible` | yes | `pages.spec.ts`, `crucible-*.spec.ts`, `discover-spread.spec.ts`, `remote-actions.spec.ts` |
+| `/settings` | yes | `pages.spec.ts` |
+| `/forge` | yes | `pages.spec.ts`, `smoke.spec.ts` |
+| `/deploy-recon` | yes | `pages.spec.ts`, `deploy-recon.spec.ts` (mocked scan) |
+| `/emberwake` | yes | `pages.spec.ts` (Cloud ecosystem panel only) |
+| `/lotl-timeline` | yes | `lotl-timeline.spec.ts`, `crucible-lotl.spec.ts` |
+| `/pathtracer` | yes | `path-tracer.spec.ts` (mocked API) |
+| `/seer`, `/oath` | yes | `pages.spec.ts` (goto; Oath not in primary nav) |
+| `/mission-deck` | yes | **None** |
+| `/builds` | yes | **None** |
+| **`/activity`** | **None** | **None** |
+| **`/roi`** | **None** | **None** |
+
+**Emberwake gaps:** no Playwright for SSM panel, policy fan-out export, spread template ZIP, launch-template genesis, or connection-test flows beyond “panel expands.”
+
+**Contingency:** no Playwright at all (only `-Contingency` Vitest/Go slice).
+
+**Court:** mocked panel only — not “fleet stuck → live court transcript on WS.”
+
+---
+
+## 4. Missing integration — multi-hop, cross-service, soak
+
+| Gap | Why it matters |
+|-----|----------------|
+| **Real multi-hop discover→spread** | Stub acknowledges 3-hop chain in UI; no agent→agent→target spread |
+| **Server process restart + Path Tracer restore** | Unit test covers new handler instance; no Playwright “start trace → restart server → resume” |
+| **Multi-agent Fleet AI scheduler** | One mocked `Tick()`; no parallel decision races (`tests/README.md`) |
+| **Cross-service cloud deploy** | Forge erasure panel → real S3 → agent `stage_fetch` magnet — never chained in CI |
+| **Deploy Recon → Crucible spread** | Links exist in UI; no E2E “scan owned host → Probe & Join same target” |
+| **Docker mining in default `test.bat`** | Separate GHA `ci-docker-mining.yml` / `scripts/ci-docker-mining.sh` — not phase 8 |
+| **Tier 3 Windows VM playbook** | `docs/E2E_VALIDATION.md` M-01–M-10 checklist — manual |
+| **Platform-split agent tests** | Linux LOTL/staging tests use `//go:build !windows` — default Windows `go test` does not execute them |
+| **WS `init` full-fleet blob** | Pagination REST-only; no load test at hundreds of agents |
+| **Hashrate retention / purge at scale** | Retention jobs exist; no sustained write soak |
+
+---
+
+## 5. Suggested additions (prioritized)
+
+### P1 — operator trust / security regressions
+
+1. **Deploy Recon SSRF canary live hit** — owned httptest target that callbacks `/recon/ping/{id}`; optional Playwright “copy URL → simulate paste field → assert WS `canary_hit`.”
+2. **Activity Feed + ROI** — minimal Vitest render + Playwright nav smoke (currently zero coverage).
+3. **`-SkipE2E` documentation trap** — treat quick verify as “no browser proof”; require phase 8 before release.
+4. **Cloud Map `route_via` server wiring** — once implemented, deploy-plan test matching agent-side expectation.
+5. **Path Tracer restart integration** — single Go test: persist → `NewPathTracerHandler` → API `status` matches (extend beyond handler-only restart).
+
+### P2 — shipped features, mock-heavy today
+
+1. **Court / contingency live-path smoke** — optional `OLLAMA_URL` job: one stuck snapshot → one court cycle → `restart_mining` dispatch (skip if no model).
+2. **Subnet recon LAN fixture** — docker-compose mini-LAN or recorded PCAP inject; prove `subnet_recon_report` → dashboard `fleetDiscoveries`.
+3. **Emberwake cloud panels Playwright** — SSM export, S3 connection test (mock AWS), policy fan-out download.
+4. **Mission Deck + Builds** — Playwright smoke + one forge→build→download happy path.
+5. **Linux spread lane CI** — run agent `deploy/...` linux-tagged tests on `ubuntu-latest` (complement Windows host).
+6. **Live forge opt-in gate** — nightly `LIVE_FORGE=1 -tags liveforge` job, not PR-blocking.
+
+### P3 — scale, polish, deferred architecture
+
+1. **Fleet soak harness** — N synthetic agents, `stats_batch` for 30+ min, assert no `SQLITE_BUSY` / memory growth.
+2. **Real multi-hop spread lab** — second VM in `E2E_VALIDATION.md` Tier 3 checklist automation.
+3. **Android emulator smoke** — install APK, auth once, scout `service_discover`.
+4. **Mesh `-tags p2p` job** — peer count > 0 in isolated network.
+5. **Path Tracer live WG** — lab-only `wg_setup` with preinstalled WireGuard (skip in CI).
+6. **Fleet topology 3D** — Vitest cap at 200 nodes (`PROBLEMS.md`); no tests today.
+
+---
+
+## Operator takeaway
+
+Passing `test.bat` means **contracts and UI wiring are well guarded**. It does **not** mean AWS spread, lateral lanes, SSRF canary confirmation, LAN subnet discovery, live forge, Android scouts, or fleet scale behave in your environment. Those are explicitly deferred to **operator IAM, lab VMs, and [`E2E_VALIDATION.md`](E2E_VALIDATION.md) Tier 2–3** — which is honest, but easy to forget because test counts look complete.
+
+If you want one command that closest approximates “real” without a Windows VM: **`ci-docker-mining`** (Linux C2 + hashrate) plus **phase 8 Playwright** (stub agents). Everything else in `PROBLEMS.md` § Manual/live remains on you.
+
+**Practical habit after landing:**
+
+1. **L2** `.\scripts\test-suite.ps1 -SkipE2E` — fast contract gate.
+2. **L3** full suite (phase 8) when UI, WS, or routing changed.
+3. Read this doc + `PROBLEMS.md` before claiming production-ready on spread, cloud, or recon.

tests/README.md

@@ -4,10 +4,11 @@
 
 ## Master validation (operator commands)
 
-**Beyond unit tests:** the [validation ladder (L0–L8)](../docs/VALIDATION_LADDER.md) maps each rung to 	est-suite.ps1 slices, Docker/VM payload steps, and honest gaps from [PROBLEMS.md](../PROBLEMS.md). Use it after parallel agent landings when you need the lowest honest gate, not another full regression table.
+**Beyond unit tests:** the [validation ladder (L0–L8)](../docs/VALIDATION_LADDER.md) maps each rung to `test-suite.ps1` slices, Docker/VM payload steps, and honest gaps from [PROBLEMS.md](../PROBLEMS.md). Use it after parallel agent landings when you need the lowest honest gate, not another full regression table.
 
+**Coverage gaps:** [`docs/TESTING_GAPS.md`](../docs/TESTING_GAPS.md) is the honest audit of what automation does *not* prove — live AWS, real spread lanes, missing Playwright routes (`/activity`, `/roi`, `/mission-deck`, `/builds`), and P1/P2/P3 test work. Treat `-SkipE2E` as contract-only; phase 8 is required before release when UI or WS changed.
 
-PROBLEMS.md `Test gaps / noise` defers here — use the tables below for P1, fleet evolution, and P2 coverage.
+PROBLEMS.md `Test gaps / noise` defers here — use the tables below for P1, fleet evolution, and P2 coverage.
 
 
 After parallel agent landings, run from repo root: